[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:33 dist-test-slave-czm5 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:33 dist-test-slave-czm5 krb5kdc[399](info): set up 2 sockets
Jun 21 05:15:33 dist-test-slave-czm5 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 05:15:34.484081 392 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:34 dist-test-slave-czm5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:34.594278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:34.670111 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:34.670360 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:34.670393 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:34.671470 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:34.671540 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:34.671563 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:34.671583 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:34.671612 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:34.672946 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:34.673197 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:34.673408 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:15:34.678822 412 server_base.cc:1034] running on GCE node
W20260621 05:15:34.678757 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:34.678830 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:34.678778 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:34.679244 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:34.679527 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:34.680678 412 hybrid_clock.cc:648] HybridClock initialized: now 1782018934680672 us; error 38 us; skew 500 ppm
Jun 21 05:15:34 dist-test-slave-czm5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:34.682924 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:34.683306 412 webserver.cc:460] Webserver started at http://127.0.98.62:45321/ using document root <none> and password file <none>
I20260621 05:15:34.683514 412 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:34.683575 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:34.683729 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:34.684738 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "e616f3ec6059493cad24bd5ae9042bdf"
format_stamp: "Formatted at 2026-06-21 05:15:34 on dist-test-slave-czm5"
server_key: "b6b5632a18f43a879723d2f82798bb6b"
server_key_iv: "e20682bad78980a7e7aeb100f93768b4"
server_key_version: "encryptionkey@0"
I20260621 05:15:34.685115 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "e616f3ec6059493cad24bd5ae9042bdf"
format_stamp: "Formatted at 2026-06-21 05:15:34 on dist-test-slave-czm5"
server_key: "b6b5632a18f43a879723d2f82798bb6b"
server_key_iv: "e20682bad78980a7e7aeb100f93768b4"
server_key_version: "encryptionkey@0"
I20260621 05:15:34.686383 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:15:34.687048 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:34.687222 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:34.687318 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "e616f3ec6059493cad24bd5ae9042bdf"
format_stamp: "Formatted at 2026-06-21 05:15:34 on dist-test-slave-czm5"
server_key: "b6b5632a18f43a879723d2f82798bb6b"
server_key_iv: "e20682bad78980a7e7aeb100f93768b4"
server_key_version: "encryptionkey@0"
I20260621 05:15:34.687391 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:34.713305 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:34.715549 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:34.715742 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:34.719537 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36839
I20260621 05:15:34.719573 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36839 every 8 connection(s)
I20260621 05:15:34.719949 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:34.720846 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:34.722890 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf: Bootstrap starting.
I20260621 05:15:34.723405 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:34.723733 480 log.cc:824] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:34.724396 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf: No bootstrap required, opened a new log
I20260621 05:15:34.725836 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e616f3ec6059493cad24bd5ae9042bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36839 } }
I20260621 05:15:34.725960 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:34.725980 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e616f3ec6059493cad24bd5ae9042bdf, State: Initialized, Role: FOLLOWER
I20260621 05:15:34.726110 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e616f3ec6059493cad24bd5ae9042bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36839 } }
I20260621 05:15:34.726197 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:34.726218 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:34.726246 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:34.726675 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e616f3ec6059493cad24bd5ae9042bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36839 } }
I20260621 05:15:34.726750 480 leader_election.cc:302] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e616f3ec6059493cad24bd5ae9042bdf; no voters:
I20260621 05:15:34.726859 480 leader_election.cc:288] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:34.726917 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:34.727062 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [term 1 LEADER]: Becoming Leader. State: Replica: e616f3ec6059493cad24bd5ae9042bdf, State: Running, Role: LEADER
I20260621 05:15:34.727119 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:34.727141 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e616f3ec6059493cad24bd5ae9042bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36839 } }
I20260621 05:15:34.727484 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e616f3ec6059493cad24bd5ae9042bdf. Latest consensus state: current_term: 1 leader_uuid: "e616f3ec6059493cad24bd5ae9042bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e616f3ec6059493cad24bd5ae9042bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36839 } } }
I20260621 05:15:34.727564 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:34.727722 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e616f3ec6059493cad24bd5ae9042bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e616f3ec6059493cad24bd5ae9042bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36839 } } }
I20260621 05:15:34.727859 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:34.727805 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:34.728519 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:34.729221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 412
I20260621 05:15:34.729326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:34.730049 392 external_mini_cluster.cc:1428] Setting key 9c9f490032de10adbd09f8d20db29141
I20260621 05:15:34.730057 490 catalog_manager.cc:1347] Generated new cluster ID: b66a6f10ee73457895b4ef82c38125a3
I20260621 05:15:34.731144 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:15:34 dist-test-slave-czm5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:34.741955 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:34.742588 490 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:34.744273 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:34.730807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.0.98.62:36839)
0621 05:15:34.731726 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:34.731734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:34.731767 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:15:34.732865 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:34.732871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:34.732919 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:34.733052 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:34.734409 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:34.735696 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:34.736737 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:34.737138 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:34.737270 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:34.740546 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:34.740551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:34.740551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:34.740563 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:34.742450 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:34.742925 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:34.742928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:34.742928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:34.743018 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:34.743291 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:34.743292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:34.743292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:34.743487 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:34.743617 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:34.743912 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 05:15:34.744066 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":72,"threads_started":1}
I20260621 05:15:34.750269 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e616f3ec6059493cad24bd5ae9042bdf: Generated new TSK 0
I20260621 05:15:34.750429 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:15:34.778246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36839
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:34.878741 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:34.878942 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:34.878963 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:34.880096 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:34.880133 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:34.880157 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:34.881490 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:34.881714 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:34.881945 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:34.887877 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:34.887871 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:34.887841 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:34.887856 506 server_base.cc:1034] running on GCE node
I20260621 05:15:34.888298 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:34.888579 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:34.889755 506 hybrid_clock.cc:648] HybridClock initialized: now 1782018934889735 us; error 35 us; skew 500 ppm
Jun 21 05:15:34 dist-test-slave-czm5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:34.892606 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:34.893043 506 webserver.cc:460] Webserver started at http://127.0.98.1:40979/ using document root <none> and password file <none>
I20260621 05:15:34.893258 506 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:34.893329 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:34.893476 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:34.894491 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a438221e6a74148bdfad5f345e849ef"
format_stamp: "Formatted at 2026-06-21 05:15:34 on dist-test-slave-czm5"
server_key: "81221850375522f0b0603eff6b230030"
server_key_iv: "d81b65bfdf761f877ec0c518fec2d120"
server_key_version: "encryptionkey@0"
I20260621 05:15:34.894853 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a438221e6a74148bdfad5f345e849ef"
format_stamp: "Formatted at 2026-06-21 05:15:34 on dist-test-slave-czm5"
server_key: "81221850375522f0b0603eff6b230030"
server_key_iv: "d81b65bfdf761f877ec0c518fec2d120"
server_key_version: "encryptionkey@0"
I20260621 05:15:34.896256 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:34.896973 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:34.897132 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:34.897228 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "9a438221e6a74148bdfad5f345e849ef"
format_stamp: "Formatted at 2026-06-21 05:15:34 on dist-test-slave-czm5"
server_key: "81221850375522f0b0603eff6b230030"
server_key_iv: "d81b65bfdf761f877ec0c518fec2d120"
server_key_version: "encryptionkey@0"
I20260621 05:15:34.897323 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:34.922353 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:34.924660 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:34.924829 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:34.925094 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:34.925412 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:34.925465 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:34.925505 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:34.925547 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:34.931486 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43545
I20260621 05:15:34.931567 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43545 every 8 connection(s)
I20260621 05:15:34.931891 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:15:34.933254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 506
I20260621 05:15:34.933372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:34.933514 392 external_mini_cluster.cc:1428] Setting key ab08327a1d7f08da9a4a14d541092a1a
Jun 21 05:15:34 dist-test-slave-czm5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260621 05:15:34.941411 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:34.932548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60315 (local address 127.0.98.62:36839)
0621 05:15:34.932588 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:34.932590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:34.932800 (+ 210us) server_negotiation.cc:407] Connection header received
0621 05:15:34.933297 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:34.933298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:34.933310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:34.933363 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:34.934491 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:34.934995 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:34.936035 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:34.936359 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:34.936422 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:34.938518 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:34.938519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:34.938520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:34.938527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:34.940083 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:34.940457 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:34.940458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:34.940459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:34.940519 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:34.940743 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:34.940745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:34.940745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:34.940885 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:34.940958 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:34.941175 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 05:15:34.941288 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:15:34.941534 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:34.932532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36839 (local address 127.0.98.1:60315)
0621 05:15:34.932661 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:34.932700 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:34.933188 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:34.933454 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:34.933458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:34.934175 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:34.934398 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:34.934403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:34.935087 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:34.935088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:34.935936 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:34.935938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:34.936444 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:34.936444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:34.936548 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:34.937057 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:34.937064 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:34.938427 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:34.940170 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:34.940176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:34.940177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:34.940385 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:34.940591 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:34.940591 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:34.940591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:34.940676 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:34.940952 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:34.940955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:34.941057 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 05:15:34.941131 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:15:34.941257 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260621 05:15:34.941980 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36839
I20260621 05:15:34.942056 635 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:34.942303 635 heartbeater.cc:511] Master 127.0.98.62:36839 requested a full tablet report, sending...
I20260621 05:15:34.942785 444 ts_manager.cc:194] Registered new tserver with Master: 9a438221e6a74148bdfad5f345e849ef (127.0.98.1:43545)
I20260621 05:15:34.943372 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:15:34.956456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36839
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:35.034963 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:35.035148 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:35.035171 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:35.036375 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:35.036427 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:35.036453 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:35.037855 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:35.038019 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:35.038257 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:35.042819 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:35.042829 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:35.042841 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:35.043341 640 server_base.cc:1034] running on GCE node
I20260621 05:15:35.043525 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:35.043823 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:35.044978 640 hybrid_clock.cc:648] HybridClock initialized: now 1782018935044966 us; error 30 us; skew 500 ppm
Jun 21 05:15:35 dist-test-slave-czm5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:35.047360 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:15:35.047777 640 webserver.cc:460] Webserver started at http://127.0.98.2:46693/ using document root <none> and password file <none>
I20260621 05:15:35.047959 640 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:35.047996 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:35.048079 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:35.048972 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab492b4d94744d14b4c53607d850a8fc"
format_stamp: "Formatted at 2026-06-21 05:15:35 on dist-test-slave-czm5"
server_key: "a41a87271417c46f47540377711c2526"
server_key_iv: "c4f239da00921cb91300cceded97a312"
server_key_version: "encryptionkey@0"
I20260621 05:15:35.049266 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab492b4d94744d14b4c53607d850a8fc"
format_stamp: "Formatted at 2026-06-21 05:15:35 on dist-test-slave-czm5"
server_key: "a41a87271417c46f47540377711c2526"
server_key_iv: "c4f239da00921cb91300cceded97a312"
server_key_version: "encryptionkey@0"
I20260621 05:15:35.050513 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:35.051196 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:35.051396 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:35.051483 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "ab492b4d94744d14b4c53607d850a8fc"
format_stamp: "Formatted at 2026-06-21 05:15:35 on dist-test-slave-czm5"
server_key: "a41a87271417c46f47540377711c2526"
server_key_iv: "c4f239da00921cb91300cceded97a312"
server_key_version: "encryptionkey@0"
I20260621 05:15:35.051577 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:35.064586 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:35.066207 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:35.066370 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:35.066573 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:35.066850 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:35.066905 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:35.066942 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:35.066968 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:35.072971 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33645
I20260621 05:15:35.073009 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33645 every 8 connection(s)
I20260621 05:15:35.073333 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:15:35 dist-test-slave-czm5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:35.080916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 640
I20260621 05:15:35.081003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:35.081168 392 external_mini_cluster.cc:1428] Setting key 8e30ad0d3e3dee456d7e295d5b360f0c
I20260621 05:15:35.081586 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:35.074078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40473 (local address 127.0.98.62:36839)
0621 05:15:35.074140 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:35.074142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:35.074309 (+ 167us) server_negotiation.cc:407] Connection header received
0621 05:15:35.074749 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:35.074750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:35.074759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:35.074812 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:35.075717 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.076196 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.077067 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.077243 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.077292 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:35.079119 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:35.079121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:35.079121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:35.079126 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:35.080262 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:35.080591 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:35.080592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:35.080592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:35.080634 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:35.080879 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:35.080880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:35.080880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:35.080965 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:35.081051 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:35.081446 (+ 395us) server_negotiation.cc:299] Negotiation successful
0621 05:15:35.081529 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:15:35.081722 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:35.074029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36839 (local address 127.0.98.2:40473)
0621 05:15:35.074187 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:35.074225 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:35.074669 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:35.074861 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:35.074864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:35.075426 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:35.075612 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:35.075616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.076270 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.076276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:35.077001 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:35.077002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.077313 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.077313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:35.077400 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:35.077804 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:35.077811 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:35.079045 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:35.080326 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:35.080331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:35.080332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:35.080524 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:35.080688 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:35.080688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:35.080688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:35.080816 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:35.081056 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:35.081061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:35.081210 (+ 149us) client_negotiation.cc:769] Sending connection context
0621 05:15:35.081335 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 05:15:35.081474 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260621 05:15:35.082217 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36839
I20260621 05:15:35.082305 769 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:35.082459 769 heartbeater.cc:511] Master 127.0.98.62:36839 requested a full tablet report, sending...
I20260621 05:15:35.082847 444 ts_manager.cc:194] Registered new tserver with Master: ab492b4d94744d14b4c53607d850a8fc (127.0.98.2:33645)
I20260621 05:15:35.083308 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40473
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:15:35.104758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36839
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:35.188689 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:35.188879 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:35.188900 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:35.190917 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:35.190970 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:35.191017 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:35.193959 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:35.194164 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:35.194330 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:35.198972 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:35.198972 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:35.199061 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:35.199223 774 server_base.cc:1034] running on GCE node
I20260621 05:15:35.199447 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:35.199746 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:35.200891 774 hybrid_clock.cc:648] HybridClock initialized: now 1782018935200869 us; error 39 us; skew 500 ppm
Jun 21 05:15:35 dist-test-slave-czm5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:35.203236 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:15:35.203735 774 webserver.cc:460] Webserver started at http://127.0.98.3:36529/ using document root <none> and password file <none>
I20260621 05:15:35.203967 774 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:35.204034 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:35.204162 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:35.205196 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5303666a9a04dea86f465d8c759d87b"
format_stamp: "Formatted at 2026-06-21 05:15:35 on dist-test-slave-czm5"
server_key: "e1d2bd94f3d8f8391cdb8aad3af081dd"
server_key_iv: "98df172b3f6759bc0de28af8a2d61d6b"
server_key_version: "encryptionkey@0"
I20260621 05:15:35.205556 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5303666a9a04dea86f465d8c759d87b"
format_stamp: "Formatted at 2026-06-21 05:15:35 on dist-test-slave-czm5"
server_key: "e1d2bd94f3d8f8391cdb8aad3af081dd"
server_key_iv: "98df172b3f6759bc0de28af8a2d61d6b"
server_key_version: "encryptionkey@0"
I20260621 05:15:35.206857 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:35.207562 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:35.207747 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:35.207856 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "a5303666a9a04dea86f465d8c759d87b"
format_stamp: "Formatted at 2026-06-21 05:15:35 on dist-test-slave-czm5"
server_key: "e1d2bd94f3d8f8391cdb8aad3af081dd"
server_key_iv: "98df172b3f6759bc0de28af8a2d61d6b"
server_key_version: "encryptionkey@0"
I20260621 05:15:35.207955 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:35.239698 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:35.241506 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:35.241671 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:35.241895 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:35.242182 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:35.242235 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:35.242281 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:35.242308 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:35.249122 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35935
I20260621 05:15:35.249157 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35935 every 8 connection(s)
I20260621 05:15:35.249476 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:15:35.249651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 774
I20260621 05:15:35.249747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:35.249920 392 external_mini_cluster.cc:1428] Setting key cbf897bed9f2d21336f1a08710daabf7
Jun 21 05:15:35 dist-test-slave-czm5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:35.259109 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:35.250167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51701 (local address 127.0.98.62:36839)
0621 05:15:35.250230 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:35.250233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:35.250521 (+ 288us) server_negotiation.cc:407] Connection header received
0621 05:15:35.251169 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:35.251170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:35.251182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:35.251239 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:35.252892 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.253352 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.254383 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.254555 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.254605 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:35.256733 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:35.256734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:35.256734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:35.256740 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:35.257989 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:35.258324 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:35.258324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:35.258325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:35.258370 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:35.258567 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:35.258568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:35.258568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:35.258710 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:35.258762 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:35.258993 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 05:15:35.259045 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 05:15:35.259248 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:35.250160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36839 (local address 127.0.98.3:51701)
0621 05:15:35.250390 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:35.250436 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:35.251073 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:35.251309 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:35.251313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:35.252637 (+ 1324us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:35.252820 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:35.252825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.253411 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.253412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:35.254317 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:35.254319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.254624 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.254625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:35.254723 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:35.255246 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:35.255262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:35.256648 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:35.258058 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:35.258063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:35.258065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:35.258261 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:35.258424 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:35.258424 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:35.258424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:35.258510 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:35.258771 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:35.258775 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:35.258877 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 05:15:35.258965 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:15:35.259070 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":70,"threads_started":1}
I20260621 05:15:35.259711 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36839
I20260621 05:15:35.259815 903 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:35.259989 903 heartbeater.cc:511] Master 127.0.98.62:36839 requested a full tablet report, sending...
I20260621 05:15:35.260370 444 ts_manager.cc:194] Registered new tserver with Master: a5303666a9a04dea86f465d8c759d87b (127.0.98.3:35935)
I20260621 05:15:35.260717 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51701
I20260621 05:15:35.261931 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:15:35 dist-test-slave-czm5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260621 05:15:35.376034 392 mini_kdc.cc:317] Time spent kinit for test-user: real 0.114s user 0.001s sys 0.002s
Jun 21 05:15:35 dist-test-slave-czm5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:15:35.387063 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:35.377178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.98.1:43545)
0621 05:15:35.379557 (+ 2379us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:35.379562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:35.379570 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:15:35.379604 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:35.379606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:35.379673 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:35.379749 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:35.380441 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.381145 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.381864 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.382175 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.382228 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:35.384438 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:35.384445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:35.384445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:35.384452 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:35.386121 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:35.386414 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:35.386417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:35.386417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:35.386480 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:35.386622 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:35.386622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:35.386623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:35.386735 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:35.386843 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:35.386893 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 05:15:35.386953 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2344,"thread_start_us":102,"threads_started":1}
W20260621 05:15:35.387496 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53832
Jun 21 05:15:35 dist-test-slave-czm5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:15:35 dist-test-slave-czm5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:15:35.480229 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:35.473144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.0.98.1:43545)
0621 05:15:35.473210 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:35.473213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:35.473307 (+ 94us) server_negotiation.cc:407] Connection header received
0621 05:15:35.473341 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:35.473342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:35.473359 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:35.473423 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:35.473997 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.474801 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.475555 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:35.475955 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:35.476023 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:35.478177 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:35.478178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:35.478178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:35.478192 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:35.479336 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:35.479658 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:35.479659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:35.479660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:35.479718 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:35.479869 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:35.479869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:35.479869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:35.479948 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:35.480015 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:35.480093 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:15:35.480156 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:15:35.480758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 506
I20260621 05:15:35.485294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 640
I20260621 05:15:35.490798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 774
I20260621 05:15:35.494915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 412
2026-06-21T05:15:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2024 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:36 dist-test-slave-czm5 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:36 dist-test-slave-czm5 krb5kdc[917](info): set up 2 sockets
Jun 21 05:15:36 dist-test-slave-czm5 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260621 05:15:37.075140 392 mini_kdc.cc:121] Time spent starting KDC: real 1.560s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:37.184034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:37.263726 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:37.263945 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:37.263979 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:37.265138 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:37.265216 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:37.265246 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:37.265277 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:37.265301 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:37.266742 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:37.267007 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:37.267244 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:37.271900 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:37.271971 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:37.272017 930 server_base.cc:1034] running on GCE node
W20260621 05:15:37.271967 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:37.272315 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:37.272624 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:37.273770 930 hybrid_clock.cc:648] HybridClock initialized: now 1782018937273753 us; error 40 us; skew 500 ppm
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:37.276197 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:37.276672 930 webserver.cc:460] Webserver started at http://127.0.98.62:38079/ using document root <none> and password file <none>
I20260621 05:15:37.276889 930 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:37.276953 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:37.277097 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:37.278072 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "ca2c8b811cfe4db9ba6333744ee01dfd"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "1a0948c4a3a7d92c8514d232127affbe"
server_key_iv: "1ad8339bab0df353272fc9c1e24ce9bd"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.278422 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca2c8b811cfe4db9ba6333744ee01dfd"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "1a0948c4a3a7d92c8514d232127affbe"
server_key_iv: "1ad8339bab0df353272fc9c1e24ce9bd"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.279721 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:15:37.280421 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.280593 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:37.280689 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "ca2c8b811cfe4db9ba6333744ee01dfd"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "1a0948c4a3a7d92c8514d232127affbe"
server_key_iv: "1ad8339bab0df353272fc9c1e24ce9bd"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.280772 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:37.307552 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:37.309288 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:37.309437 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:37.313318 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34883
I20260621 05:15:37.313356 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34883 every 8 connection(s)
I20260621 05:15:37.313704 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:37.314661 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:37.316850 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd: Bootstrap starting.
I20260621 05:15:37.317343 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:37.317569 998 log.cc:824] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:37.317934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 930
I20260621 05:15:37.318023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:37.318187 392 external_mini_cluster.cc:1428] Setting key 302362ee898df306af3ef8183850d594
I20260621 05:15:37.318243 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd: No bootstrap required, opened a new log
I20260621 05:15:37.320448 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c8b811cfe4db9ba6333744ee01dfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } }
I20260621 05:15:37.320600 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:37.320623 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2c8b811cfe4db9ba6333744ee01dfd, State: Initialized, Role: FOLLOWER
I20260621 05:15:37.320748 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c8b811cfe4db9ba6333744ee01dfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } }
I20260621 05:15:37.320822 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:37.320878 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:37.320923 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:37.321440 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c8b811cfe4db9ba6333744ee01dfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } }
I20260621 05:15:37.321537 998 leader_election.cc:302] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2c8b811cfe4db9ba6333744ee01dfd; no voters:
I20260621 05:15:37.321732 998 leader_election.cc:288] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:37.321762 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:37.321909 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [term 1 LEADER]: Becoming Leader. State: Replica: ca2c8b811cfe4db9ba6333744ee01dfd, State: Running, Role: LEADER
I20260621 05:15:37.321982 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:37.322041 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c8b811cfe4db9ba6333744ee01dfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } }
I20260621 05:15:37.322427 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2c8b811cfe4db9ba6333744ee01dfd. Latest consensus state: current_term: 1 leader_uuid: "ca2c8b811cfe4db9ba6333744ee01dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c8b811cfe4db9ba6333744ee01dfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } } }
I20260621 05:15:37.322430 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2c8b811cfe4db9ba6333744ee01dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c8b811cfe4db9ba6333744ee01dfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } } }
I20260621 05:15:37.322605 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:37.322615 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:37.322840 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:37.323421 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:37.325219 1012 catalog_manager.cc:1347] Generated new cluster ID: 023e7fa27aa74b6390e474886f50e9ef
I20260621 05:15:37.325271 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:37.329725 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.318658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.0.98.62:34883)
0621 05:15:37.318815 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.318824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.318857 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:15:37.319454 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.319461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.319506 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.319610 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:37.320362 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.321730 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.322732 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.323354 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.323487 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.326155 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:37.326159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:37.326159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:37.326171 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:37.328160 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.328524 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.328527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.328528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.328630 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.328820 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.328821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.328821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.329130 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:37.329272 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.329349 (+ 77us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.329470 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260621 05:15:37.332204 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:37.332638 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:37.336961 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2c8b811cfe4db9ba6333744ee01dfd: Generated new TSK 0
I20260621 05:15:37.337105 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:15:37.365855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34883
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:37.445458 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:37.445772 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:37.445828 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:37.447800 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:37.447885 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:37.447954 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:37.449390 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:37.449618 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:37.449841 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:37.454475 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:37.454500 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:37.454734 1024 server_base.cc:1034] running on GCE node
W20260621 05:15:37.454475 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:37.454962 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:37.455138 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:37.456264 1024 hybrid_clock.cc:648] HybridClock initialized: now 1782018937456248 us; error 32 us; skew 500 ppm
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:37.458490 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:37.458884 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36183/ using document root <none> and password file <none>
I20260621 05:15:37.459136 1024 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:37.459193 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:37.459316 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:37.460319 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "747df1ca04284eeab96e2d2c41fdf9d8"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "7b81b256f0a5eeae68514ba7fa48b7a7"
server_key_iv: "b5fefdcbff69e9cb1610e7a1d4504b02"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.460645 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "747df1ca04284eeab96e2d2c41fdf9d8"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "7b81b256f0a5eeae68514ba7fa48b7a7"
server_key_iv: "b5fefdcbff69e9cb1610e7a1d4504b02"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.461855 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:15:37.462530 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.462690 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:37.462764 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "747df1ca04284eeab96e2d2c41fdf9d8"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "7b81b256f0a5eeae68514ba7fa48b7a7"
server_key_iv: "b5fefdcbff69e9cb1610e7a1d4504b02"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.462815 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:37.473985 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:37.475675 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:37.475790 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:37.475965 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:37.476187 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:37.476215 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.476234 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:37.476248 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.481683 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44087
I20260621 05:15:37.481743 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44087 every 8 connection(s)
I20260621 05:15:37.481954 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:37.489928 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.482558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53367 (local address 127.0.98.62:34883)
0621 05:15:37.482599 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.482602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.482822 (+ 220us) server_negotiation.cc:407] Connection header received
0621 05:15:37.483264 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.483264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.483273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.483321 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:37.484259 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.484767 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.485672 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.485857 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.485905 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.487800 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:37.487801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:37.487802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:37.487807 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:37.488889 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.489213 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.489214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.489214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.489257 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.489453 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.489453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.489453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.489539 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:37.489585 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.489809 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.489868 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:15:37.490083 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.482546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34883 (local address 127.0.98.1:53367)
0621 05:15:37.482710 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:37.482742 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:37.483189 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:37.483400 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:37.483403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:37.483990 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:37.484165 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.484170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.484839 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.484840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.485593 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.485595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.485926 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.485927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.486017 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.486412 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:37.486420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:37.487723 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:37.488954 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:37.488959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:37.488960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:37.489152 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:37.489319 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:37.489319 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:37.489320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:37.489403 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:37.489590 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:37.489593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:37.489709 (+ 116us) client_negotiation.cc:769] Sending connection context
0621 05:15:37.489785 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:15:37.489896 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260621 05:15:37.490280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 1024
I20260621 05:15:37.490374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:37.490537 392 external_mini_cluster.cc:1428] Setting key 51ab987cda8fc484427b618dd0629d8d
I20260621 05:15:37.490564 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34883
I20260621 05:15:37.490650 1153 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:37.490839 1153 heartbeater.cc:511] Master 127.0.98.62:34883 requested a full tablet report, sending...
I20260621 05:15:37.491346 962 ts_manager.cc:194] Registered new tserver with Master: 747df1ca04284eeab96e2d2c41fdf9d8 (127.0.98.1:44087)
I20260621 05:15:37.492206 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53367
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:15:37.512331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34883
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:37.592645 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:37.592861 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:37.592897 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:37.593997 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:37.594061 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:37.594142 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:37.595541 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:37.595808 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:37.596029 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:37.600649 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:37.600692 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:37.600649 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:37.600835 1158 server_base.cc:1034] running on GCE node
I20260621 05:15:37.601085 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:37.601306 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:37.602468 1158 hybrid_clock.cc:648] HybridClock initialized: now 1782018937602441 us; error 46 us; skew 500 ppm
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:37.604869 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:15:37.605280 1158 webserver.cc:460] Webserver started at http://127.0.98.2:41597/ using document root <none> and password file <none>
I20260621 05:15:37.605523 1158 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:37.605568 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:37.605726 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:37.606777 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "860deec21b2349eb9d1f168a74c12f5b"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "f44e97de171147e857fdef60b5e90c05"
server_key_iv: "278ecd693e15a473008e7a1da8990437"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.607115 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "860deec21b2349eb9d1f168a74c12f5b"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "f44e97de171147e857fdef60b5e90c05"
server_key_iv: "278ecd693e15a473008e7a1da8990437"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.608430 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:15:37.609112 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.609290 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.609383 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "860deec21b2349eb9d1f168a74c12f5b"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "f44e97de171147e857fdef60b5e90c05"
server_key_iv: "278ecd693e15a473008e7a1da8990437"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.609447 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:37.632895 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:37.634690 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:37.634859 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:37.635088 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:37.635396 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:37.635450 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.635491 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:37.635520 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.640954 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46229
I20260621 05:15:37.641037 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46229 every 8 connection(s)
I20260621 05:15:37.641320 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:15:37.646600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 1158
I20260621 05:15:37.646698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:37.646867 392 external_mini_cluster.cc:1428] Setting key de64bdf43d3b6dc27dd7c54a9fc3262f
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:37.650450 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.642010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55303 (local address 127.0.98.62:34883)
0621 05:15:37.642063 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.642064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.642246 (+ 182us) server_negotiation.cc:407] Connection header received
0621 05:15:37.642719 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.642720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.642733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.642769 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:37.643582 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.644076 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.645085 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.645288 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.645374 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.647467 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:37.647469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:37.647469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:37.647477 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:37.649137 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.649530 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.649531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.649532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.649605 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.649840 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.649841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.649841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.649953 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:37.650027 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.650293 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.650364 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:15:37.650861 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.641979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34883 (local address 127.0.98.2:55303)
0621 05:15:37.642142 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:37.642172 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:37.642619 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:37.642828 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:37.642837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:37.643320 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:37.643487 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.643490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.644181 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.644182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.644993 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.644995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.645382 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.645383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.645499 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.645901 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:37.645908 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:37.647363 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:37.649226 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:37.649231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:37.649233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:37.649453 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:37.649670 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:37.649670 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:37.649670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:37.649760 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:37.650017 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:37.650022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:37.650155 (+ 133us) client_negotiation.cc:769] Sending connection context
0621 05:15:37.650269 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 05:15:37.650415 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260621 05:15:37.651337 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34883
I20260621 05:15:37.651423 1287 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:37.651604 1287 heartbeater.cc:511] Master 127.0.98.62:34883 requested a full tablet report, sending...
I20260621 05:15:37.652051 962 ts_manager.cc:194] Registered new tserver with Master: 860deec21b2349eb9d1f168a74c12f5b (127.0.98.2:46229)
I20260621 05:15:37.652385 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55303
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:15:37.670181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34883
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:37.747604 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:37.747814 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:37.747838 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:37.748890 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:37.748922 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:37.748947 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:37.750195 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:37.750362 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:37.750514 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:15:37.755116 1292 server_base.cc:1034] running on GCE node
W20260621 05:15:37.755168 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:37.755223 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:37.755141 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:37.755505 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:37.755753 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:37.756889 1292 hybrid_clock.cc:648] HybridClock initialized: now 1782018937756873 us; error 39 us; skew 500 ppm
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:37.759109 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:15:37.759547 1292 webserver.cc:460] Webserver started at http://127.0.98.3:40161/ using document root <none> and password file <none>
I20260621 05:15:37.759770 1292 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:37.759819 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:37.759908 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:37.760847 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "5965848b1a8d4620bec9650b561d3347"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "dc6ab81436b749b94c46998185a0bf0d"
server_key_iv: "10582faa820f9b87bb3e224f00bec7a9"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.761133 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5965848b1a8d4620bec9650b561d3347"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "dc6ab81436b749b94c46998185a0bf0d"
server_key_iv: "10582faa820f9b87bb3e224f00bec7a9"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.762285 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:15:37.762889 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.763087 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:37.763181 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "5965848b1a8d4620bec9650b561d3347"
format_stamp: "Formatted at 2026-06-21 05:15:37 on dist-test-slave-czm5"
server_key: "dc6ab81436b749b94c46998185a0bf0d"
server_key_iv: "10582faa820f9b87bb3e224f00bec7a9"
server_key_version: "encryptionkey@0"
I20260621 05:15:37.763265 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:37.777279 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:37.779367 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:37.779515 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:37.779762 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:37.780058 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:37.780108 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.780146 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:37.780172 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:37.785912 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45399
I20260621 05:15:37.785961 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45399 every 8 connection(s)
I20260621 05:15:37.786245 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:37.794225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 1292
I20260621 05:15:37.794332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:37.794507 392 external_mini_cluster.cc:1428] Setting key f640923e1c9d6393666cb3abaf8a9527
I20260621 05:15:37.794840 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.786856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37825 (local address 127.0.98.62:34883)
0621 05:15:37.786922 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.786924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.787146 (+ 222us) server_negotiation.cc:407] Connection header received
0621 05:15:37.787594 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.787595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.787605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.787729 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:37.788576 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.789067 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.789980 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.790170 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.790224 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.792159 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:37.792160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:37.792160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:37.792165 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:37.793429 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.793773 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.793774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.793774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.793836 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.794058 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.794059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.794059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.794261 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:37.794346 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.794663 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.794738 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:15:37.795001 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.786838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34883 (local address 127.0.98.3:37825)
0621 05:15:37.787034 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:37.787067 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:37.787518 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:37.787783 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:37.787785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:37.788332 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:37.788502 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.788506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.789142 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.789142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.789909 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.789910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.790234 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.790234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.790319 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.790785 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:37.790792 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:37.792087 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:37.793500 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:37.793505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:37.793507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:37.793701 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:37.793893 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:37.793893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:37.793894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:37.793979 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:37.794347 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:37.794351 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:37.794490 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 05:15:37.794634 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 05:15:37.794782 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260621 05:15:37.795475 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34883
I20260621 05:15:37.795549 1421 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:37.795739 1421 heartbeater.cc:511] Master 127.0.98.62:34883 requested a full tablet report, sending...
I20260621 05:15:37.796075 962 ts_manager.cc:194] Registered new tserver with Master: 5965848b1a8d4620bec9650b561d3347 (127.0.98.3:45399)
I20260621 05:15:37.796386 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37825
I20260621 05:15:37.806228 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:15:37.813265 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.807751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.62:34883)
0621 05:15:37.807816 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.807818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.807888 (+ 70us) server_negotiation.cc:407] Connection header received
0621 05:15:37.807924 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.807925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.807933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.807973 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:37.808478 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.808923 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.809582 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.809749 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.809797 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.810832 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:37.810833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:37.810834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:37.810839 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:37.812275 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.812558 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.812558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.812559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.812604 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.812748 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.812748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.812749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.812875 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:37.812974 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.813112 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.813170 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:15:37.815174 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:15:37.816758 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:37.822914 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.818985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.0.98.1:44087)
0621 05:15:37.819113 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.819118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.819130 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:15:37.819169 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.819171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.819186 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.819273 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:37.819515 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.820375 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.821823 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.822445 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.822530 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.822588 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.822782 (+ 194us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.822823 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260621 05:15:37.822978 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.818793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44087 (local address 127.0.0.1:50456)
0621 05:15:37.818950 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:37.818986 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:37.819044 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:37.819375 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:37.819379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:37.819381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:37.819438 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.819439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.820473 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.820474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.821728 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.821730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.822529 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.822530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.822650 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.822664 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:15:37.822767 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 05:15:37.822897 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":39,"thread_start_us":62,"threads_started":1}
I20260621 05:15:37.823242 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.818878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.0.98.3:45399)
0621 05:15:37.819149 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.819153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.819191 (+ 38us) server_negotiation.cc:407] Connection header received
0621 05:15:37.819260 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.819262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.819275 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.819350 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:37.819537 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.820700 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.822000 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.822777 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.822832 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.822884 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.823125 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.823163 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260621 05:15:37.823261 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.818793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45399 (local address 127.0.0.1:53504)
0621 05:15:37.819142 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:37.819150 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:37.819200 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:37.819378 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:37.819379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:37.819384 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:37.819452 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.819454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.820834 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.820835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.821939 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.821941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.822913 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.822914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.822969 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.822971 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:15:37.823029 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:15:37.823086 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":62,"threads_started":1}
I20260621 05:15:37.823513 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.818981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46229 (local address 127.0.0.1:47826)
0621 05:15:37.819392 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:37.819401 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:37.819452 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:37.819669 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:37.819670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:37.819672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:37.819712 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.819714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.821010 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.821010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.821836 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.821838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.822824 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.822825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.822883 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.822889 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:15:37.822931 (+ 42us) client_negotiation.cc:240] Negotiation successful
0621 05:15:37.823425 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":34,"threads_started":1}
I20260621 05:15:37.823716 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.819151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.0.98.2:46229)
0621 05:15:37.819350 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.819355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.819506 (+ 151us) server_negotiation.cc:407] Connection header received
0621 05:15:37.819547 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.819550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.819564 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.819661 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:37.819901 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.820908 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.821918 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.822710 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.823082 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.823141 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.823163 (+ 22us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.823196 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260621 05:15:37.823939 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 84c97e907c33419dbb97ef9acef53601 (DEFAULT_TABLE table=test-table [id=12c44de7df9046dcb9c2a61c36166e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:37.824010 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 84c97e907c33419dbb97ef9acef53601 (DEFAULT_TABLE table=test-table [id=12c44de7df9046dcb9c2a61c36166e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:37.824339 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c97e907c33419dbb97ef9acef53601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:37.824360 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c97e907c33419dbb97ef9acef53601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:37.824537 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 84c97e907c33419dbb97ef9acef53601 (DEFAULT_TABLE table=test-table [id=12c44de7df9046dcb9c2a61c36166e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:37.824836 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c97e907c33419dbb97ef9acef53601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:37.826840 1435 tablet_bootstrap.cc:492] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347: Bootstrap starting.
I20260621 05:15:37.827443 1435 tablet_bootstrap.cc:654] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:37.828053 1436 tablet_bootstrap.cc:492] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8: Bootstrap starting.
I20260621 05:15:37.828086 1435 log.cc:824] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:37.828852 1436 tablet_bootstrap.cc:654] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:37.828898 1435 tablet_bootstrap.cc:492] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347: No bootstrap required, opened a new log
I20260621 05:15:37.828962 1435 ts_tablet_manager.cc:1397] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:15:37.829387 1436 log.cc:824] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:37.830075 1436 tablet_bootstrap.cc:492] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8: No bootstrap required, opened a new log
I20260621 05:15:37.830139 1436 ts_tablet_manager.cc:1397] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:37.830925 1435 raft_consensus.cc:348] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.831115 1435 raft_consensus.cc:374] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:37.831151 1435 raft_consensus.cc:729] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5965848b1a8d4620bec9650b561d3347, State: Initialized, Role: FOLLOWER
I20260621 05:15:37.831290 1435 consensus_queue.cc:260] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.831552 1435 ts_tablet_manager.cc:1428] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:15:37.831585 1421 heartbeater.cc:503] Master 127.0.98.62:34883 was elected leader, sending a full tablet report...
I20260621 05:15:37.831694 1438 tablet_bootstrap.cc:492] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b: Bootstrap starting.
I20260621 05:15:37.832163 1436 raft_consensus.cc:348] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.832311 1438 tablet_bootstrap.cc:654] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:37.832343 1436 raft_consensus.cc:374] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:37.832398 1436 raft_consensus.cc:729] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747df1ca04284eeab96e2d2c41fdf9d8, State: Initialized, Role: FOLLOWER
I20260621 05:15:37.832549 1436 consensus_queue.cc:260] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.832855 1436 ts_tablet_manager.cc:1428] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:15:37.832871 1438 log.cc:824] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:37.832890 1153 heartbeater.cc:503] Master 127.0.98.62:34883 was elected leader, sending a full tablet report...
I20260621 05:15:37.833556 1438 tablet_bootstrap.cc:492] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b: No bootstrap required, opened a new log
I20260621 05:15:37.833642 1438 ts_tablet_manager.cc:1397] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260621 05:15:37.835490 1438 raft_consensus.cc:348] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.835613 1438 raft_consensus.cc:374] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:37.835726 1438 raft_consensus.cc:729] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860deec21b2349eb9d1f168a74c12f5b, State: Initialized, Role: FOLLOWER
I20260621 05:15:37.835824 1438 consensus_queue.cc:260] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.836006 1438 ts_tablet_manager.cc:1428] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:15:37.836067 1287 heartbeater.cc:503] Master 127.0.98.62:34883 was elected leader, sending a full tablet report...
W20260621 05:15:37.891950 1288 tablet.cc:2369] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:15:37.934862 1440 raft_consensus.cc:482] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:37.935021 1440 raft_consensus.cc:504] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.935424 1440 leader_election.cc:288] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 747df1ca04284eeab96e2d2c41fdf9d8 (127.0.98.1:44087), 860deec21b2349eb9d1f168a74c12f5b (127.0.98.2:46229)
I20260621 05:15:37.938480 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.935613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44087 (local address 127.0.98.3:58527)
0621 05:15:37.935697 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:37.935710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:37.935759 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:37.935949 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:37.935950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:37.935952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:37.936030 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.936032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.936778 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.936779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.937702 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.937704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.938227 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.938228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.938302 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.938306 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:37.938357 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:15:37.938387 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":81}
I20260621 05:15:37.938494 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.935744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58527 (local address 127.0.98.1:44087)
0621 05:15:37.935790 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.935791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.935800 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:15:37.935828 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.935830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.935839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.935923 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:37.936109 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.936684 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.937822 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.938150 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.938202 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.938210 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.938383 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.938431 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:15:37.938567 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.935706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54397 (local address 127.0.98.2:46229)
0621 05:15:37.935768 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.935770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.935937 (+ 167us) server_negotiation.cc:407] Connection header received
0621 05:15:37.935967 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.935968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.935981 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.936042 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:37.936215 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.936721 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.937793 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.938174 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.938263 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.938272 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.938449 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.938488 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:15:37.938593 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.935658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46229 (local address 127.0.98.3:54397)
0621 05:15:37.935842 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:37.935852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:37.935905 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:37.936054 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:37.936055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:37.936056 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:37.936148 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.936150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.936810 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.936811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.937704 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:37.937706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.938287 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.938288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:37.938367 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.938371 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:37.938427 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:15:37.938459 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":13,"thread_start_us":98,"threads_started":1}
I20260621 05:15:37.938848 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c97e907c33419dbb97ef9acef53601" candidate_uuid: "5965848b1a8d4620bec9650b561d3347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" is_pre_election: true
I20260621 05:15:37.938974 1107 raft_consensus.cc:2454] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5965848b1a8d4620bec9650b561d3347 in term 0.
I20260621 05:15:37.939338 1309 leader_election.cc:302] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5965848b1a8d4620bec9650b561d3347, 747df1ca04284eeab96e2d2c41fdf9d8; no voters:
I20260621 05:15:37.939424 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c97e907c33419dbb97ef9acef53601" candidate_uuid: "5965848b1a8d4620bec9650b561d3347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "860deec21b2349eb9d1f168a74c12f5b" is_pre_election: true
I20260621 05:15:37.939476 1440 raft_consensus.cc:2781] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:37.939527 1440 raft_consensus.cc:482] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:37.939551 1440 raft_consensus.cc:3037] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:37.939565 1241 raft_consensus.cc:2454] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5965848b1a8d4620bec9650b561d3347 in term 0.
I20260621 05:15:37.940290 1440 raft_consensus.cc:504] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.940505 1440 leader_election.cc:288] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [CANDIDATE]: Term 1 election: Requested vote from peers 747df1ca04284eeab96e2d2c41fdf9d8 (127.0.98.1:44087), 860deec21b2349eb9d1f168a74c12f5b (127.0.98.2:46229)
I20260621 05:15:37.940580 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c97e907c33419dbb97ef9acef53601" candidate_uuid: "5965848b1a8d4620bec9650b561d3347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747df1ca04284eeab96e2d2c41fdf9d8"
I20260621 05:15:37.940658 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c97e907c33419dbb97ef9acef53601" candidate_uuid: "5965848b1a8d4620bec9650b561d3347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "860deec21b2349eb9d1f168a74c12f5b"
I20260621 05:15:37.940667 1107 raft_consensus.cc:3037] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:37.940732 1241 raft_consensus.cc:3037] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:37.941555 1241 raft_consensus.cc:2454] T 84c97e907c33419dbb97ef9acef53601 P 860deec21b2349eb9d1f168a74c12f5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5965848b1a8d4620bec9650b561d3347 in term 1.
I20260621 05:15:37.941555 1107 raft_consensus.cc:2454] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5965848b1a8d4620bec9650b561d3347 in term 1.
I20260621 05:15:37.941757 1308 leader_election.cc:302] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5965848b1a8d4620bec9650b561d3347, 860deec21b2349eb9d1f168a74c12f5b; no voters:
I20260621 05:15:37.941859 1440 raft_consensus.cc:2781] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:37.942008 1440 raft_consensus.cc:686] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [term 1 LEADER]: Becoming Leader. State: Replica: 5965848b1a8d4620bec9650b561d3347, State: Running, Role: LEADER
I20260621 05:15:37.942108 1440 consensus_queue.cc:237] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } }
I20260621 05:15:37.942879 962 catalog_manager.cc:5495] T 84c97e907c33419dbb97ef9acef53601 P 5965848b1a8d4620bec9650b561d3347 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5965848b1a8d4620bec9650b561d3347 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5965848b1a8d4620bec9650b561d3347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747df1ca04284eeab96e2d2c41fdf9d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5965848b1a8d4620bec9650b561d3347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "860deec21b2349eb9d1f168a74c12f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46229 } health_report { overall_health: UNKNOWN } } }
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:15:37.975059 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.967423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.98.1:44087)
0621 05:15:37.967470 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.967472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.967637 (+ 165us) server_negotiation.cc:407] Connection header received
0621 05:15:37.967730 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.967731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.967741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.967828 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:37.968375 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.968828 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.969540 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.969722 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.969810 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.972112 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:37.972123 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:37.972124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:37.972131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:37.973457 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.973841 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.973844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.973845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.974305 (+ 460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.974513 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.974514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.974514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.974654 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:37.974827 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.974868 (+ 41us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.974947 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260621 05:15:37.982590 1154 tablet.cc:2369] T 84c97e907c33419dbb97ef9acef53601 P 747df1ca04284eeab96e2d2c41fdf9d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:15:37 dist-test-slave-czm5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:15:37.996807 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:37.990027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.0.98.1:44087)
0621 05:15:37.990103 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:37.990105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:37.990208 (+ 103us) server_negotiation.cc:407] Connection header received
0621 05:15:37.990283 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:37.990284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:37.990295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:37.990359 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:37.990878 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.991322 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.992105 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:37.992282 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:37.992351 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:37.994676 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:37.994677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:37.994677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:37.994683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:37.995878 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.996167 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.996168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.996168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.996232 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:37.996382 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:37.996382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:37.996382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:37.996459 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:37.996526 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:37.996647 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 05:15:37.996702 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 05:15:38 dist-test-slave-czm5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:15:38 dist-test-slave-czm5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:15:38.017602 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:38.011427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.0.98.1:44087)
0621 05:15:38.011499 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:38.011504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:38.011570 (+ 66us) server_negotiation.cc:407] Connection header received
0621 05:15:38.011608 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:38.011608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:38.011649 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:38.011707 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:38.012184 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:38.012656 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:38.013372 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:38.013532 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:38.013579 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:38.015499 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:38.015500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:38.015500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:38.015506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:38.016743 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:38.017041 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:38.017042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:38.017042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:38.017086 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:38.017245 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:38.017245 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:38.017245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:38.017322 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:38.017389 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:38.017469 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 05:15:38.017516 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260621 05:15:38.018059 1067 tablet_service.cc:2908] Rejecting scan request for tablet 84c97e907c33419dbb97ef9acef53601: Uninitialized: clean time has not yet been initialized
I20260621 05:15:38.018869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 1024
I20260621 05:15:38.023690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 1158
I20260621 05:15:38.028909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 1292
I20260621 05:15:38.033190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 930
2026-06-21T05:15:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2539 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:38 dist-test-slave-czm5 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:38 dist-test-slave-czm5 krb5kdc[1459](info): set up 2 sockets
Jun 21 05:15:38 dist-test-slave-czm5 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260621 05:15:39.047199 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:39.156081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:39.234123 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:39.234426 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:39.234465 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:39.236030 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:39.236073 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:15:39.236086 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:39.236099 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:39.236111 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:39.236123 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:39.237499 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45145
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:39.237672 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:39.237836 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:39.242515 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:39.242645 1472 server_base.cc:1034] running on GCE node
W20260621 05:15:39.242578 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:39.242511 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:39.243017 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:39.243286 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:39.244451 1472 hybrid_clock.cc:648] HybridClock initialized: now 1782018939244437 us; error 47 us; skew 500 ppm
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:39.249766 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:39.250182 1472 webserver.cc:460] Webserver started at http://127.0.98.62:33169/ using document root <none> and password file <none>
I20260621 05:15:39.250393 1472 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:39.250456 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:39.250591 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:39.251573 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "d194049e78ef43e0bf7f2d2be3b6e217"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "ba65c99fac13a2ea7d0c726f6e5d7a98"
server_key_iv: "79642c033ac64ecdf100cbcc75a3e96e"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.251973 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "d194049e78ef43e0bf7f2d2be3b6e217"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "ba65c99fac13a2ea7d0c726f6e5d7a98"
server_key_iv: "79642c033ac64ecdf100cbcc75a3e96e"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.253192 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:39.253840 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.254001 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:39.254089 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "d194049e78ef43e0bf7f2d2be3b6e217"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "ba65c99fac13a2ea7d0c726f6e5d7a98"
server_key_iv: "79642c033ac64ecdf100cbcc75a3e96e"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.254181 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:39.262972 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:39.265169 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:39.265327 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:39.269667 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45145
I20260621 05:15:39.269706 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45145 every 8 connection(s)
I20260621 05:15:39.270000 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:39.270428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 1472
I20260621 05:15:39.270527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:39.270696 392 external_mini_cluster.cc:1428] Setting key 904fe3b5863988c057265845447750b2
I20260621 05:15:39.270697 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:39.273492 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217: Bootstrap starting.
I20260621 05:15:39.274082 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:39.274377 1550 log.cc:824] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:39.275010 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217: No bootstrap required, opened a new log
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:39.276785 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d194049e78ef43e0bf7f2d2be3b6e217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } }
I20260621 05:15:39.276985 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:39.277035 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d194049e78ef43e0bf7f2d2be3b6e217, State: Initialized, Role: FOLLOWER
I20260621 05:15:39.277169 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d194049e78ef43e0bf7f2d2be3b6e217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } }
I20260621 05:15:39.277228 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:39.277264 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:39.277345 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:39.278019 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d194049e78ef43e0bf7f2d2be3b6e217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } }
I20260621 05:15:39.278134 1550 leader_election.cc:302] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d194049e78ef43e0bf7f2d2be3b6e217; no voters:
I20260621 05:15:39.278281 1550 leader_election.cc:288] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:39.278328 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:39.278475 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [term 1 LEADER]: Becoming Leader. State: Replica: d194049e78ef43e0bf7f2d2be3b6e217, State: Running, Role: LEADER
I20260621 05:15:39.278607 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:39.278640 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d194049e78ef43e0bf7f2d2be3b6e217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } }
I20260621 05:15:39.279049 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d194049e78ef43e0bf7f2d2be3b6e217. Latest consensus state: current_term: 1 leader_uuid: "d194049e78ef43e0bf7f2d2be3b6e217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d194049e78ef43e0bf7f2d2be3b6e217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } } }
I20260621 05:15:39.279196 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:39.279403 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d194049e78ef43e0bf7f2d2be3b6e217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d194049e78ef43e0bf7f2d2be3b6e217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } } }
I20260621 05:15:39.279488 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:39.280277 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:39.281009 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.271237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.0.98.62:45145)
0621 05:15:39.271425 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.271433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.271457 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:15:39.272036 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.272043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.272088 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.272238 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:39.272817 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.273764 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.274609 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.275009 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.275139 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.277356 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:39.277359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:39.277359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:39.277368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:39.279363 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.279850 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.279853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.279853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.279958 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.280170 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.280171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.280172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.280476 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:39.280580 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.280698 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.280825 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260621 05:15:39.281272 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:39.282644 1564 catalog_manager.cc:1347] Generated new cluster ID: 1abd8bf5e79e4f38ac275fff07b097e7
I20260621 05:15:39.282692 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:39.295287 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:39.295830 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:39.298648 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217: Generated new TSK 0
I20260621 05:15:39.298810 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:15:39.324455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:39.418696 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:39.418879 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:39.418901 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:39.420048 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:39.420086 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:39.420106 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:15:39.420122 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:39.421535 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:39.421749 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:39.422008 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:39.426779 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:39.426779 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:39.426970 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:39.426795 1576 server_base.cc:1034] running on GCE node
I20260621 05:15:39.427173 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:39.427443 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:39.428602 1576 hybrid_clock.cc:648] HybridClock initialized: now 1782018939428567 us; error 51 us; skew 500 ppm
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:39.430891 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:39.431294 1576 webserver.cc:460] Webserver started at http://127.0.98.1:36577/ using document root <none> and password file <none>
I20260621 05:15:39.431577 1576 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:39.431670 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:39.431814 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:39.432838 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "52da5ece527b420bad61cf43836b8cea"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "02030c26418c41ae558eb64473964e68"
server_key_iv: "91e39c3a34efa51b76a8df3b6c718f32"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.433188 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "52da5ece527b420bad61cf43836b8cea"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "02030c26418c41ae558eb64473964e68"
server_key_iv: "91e39c3a34efa51b76a8df3b6c718f32"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.434466 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:39.435117 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.435278 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:39.435372 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "52da5ece527b420bad61cf43836b8cea"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "02030c26418c41ae558eb64473964e68"
server_key_iv: "91e39c3a34efa51b76a8df3b6c718f32"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.435447 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:39.446539 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:39.448338 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:39.448483 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:39.448987 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:39.449048 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.449090 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:39.449126 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.455741 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40023
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:39.455811 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40023 every 8 connection(s)
I20260621 05:15:39.456151 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:15:39.459237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 1576
I20260621 05:15:39.459332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:39.459384 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.449421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33471 (local address 127.0.98.62:45145)
0621 05:15:39.449480 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.449482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.449755 (+ 273us) server_negotiation.cc:407] Connection header received
0621 05:15:39.450209 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.450210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.450223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.450304 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:39.451123 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.451954 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.453491 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.453683 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.453755 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.456474 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:39.456475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:39.456476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:39.456484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:39.458140 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.458499 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.458500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.458500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.458548 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.458749 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.458749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.458750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.458849 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:39.458897 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.459173 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.459258 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:15:39.459515 392 external_mini_cluster.cc:1428] Setting key 2829260c6ba66b847fa49c6e59bc6442
I20260621 05:15:39.459511 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.449423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.1:33471)
0621 05:15:39.449602 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.449639 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.450124 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.450387 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.450391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.450747 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:39.451019 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.451025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.452047 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.452048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.453369 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.453372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.453748 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.453749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.453897 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.454389 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:39.454401 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:39.456357 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:39.458207 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.458212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.458214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.458432 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.458605 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.458606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.458606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.458697 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.458903 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:39.458907 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:39.459030 (+ 123us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.459130 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.459248 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260621 05:15:39.460000 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45145
I20260621 05:15:39.460078 1707 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:39.460259 1707 heartbeater.cc:511] Master 127.0.98.62:45145 requested a full tablet report, sending...
I20260621 05:15:39.460880 1503 ts_manager.cc:194] Registered new tserver with Master: 52da5ece527b420bad61cf43836b8cea (127.0.98.1:40023)
I20260621 05:15:39.461807 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33471
I20260621 05:15:39.467089 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.461035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58765 (local address 127.0.98.62:45145)
0621 05:15:39.461071 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.461073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.461239 (+ 166us) server_negotiation.cc:407] Connection header received
0621 05:15:39.461311 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.461312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.461328 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.461383 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:39.461738 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.462348 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.463339 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.463507 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.463560 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.464620 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:39.464621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:39.464622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:39.464627 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:39.466032 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.466424 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.466424 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.466425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.466482 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.466666 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.466667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.466667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.466762 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:39.466806 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.466973 (+ 167us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.467028 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:15:39.467187 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.461002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.1:58765)
0621 05:15:39.461166 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.461176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.461254 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.461387 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.461388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.461560 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:39.461630 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.461632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.462443 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.462444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.463257 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.463259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.463581 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.463582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.463685 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.464090 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:39.464095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:39.464538 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:39.466124 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.466125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.466125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.466341 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.466544 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.466545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.466545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.466610 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.466829 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:39.466830 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:39.466878 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.466958 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.467065 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:15:39.485806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:39.567075 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:39.567344 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:39.567386 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:39.568624 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:39.568694 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:39.568727 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:15:39.568765 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:39.570277 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:39.570557 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:39.570797 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:39.575788 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:39.575868 1716 server_base.cc:1034] running on GCE node
W20260621 05:15:39.575795 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:39.575798 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:39.576248 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:39.576494 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:39.577656 1716 hybrid_clock.cc:648] HybridClock initialized: now 1782018939577639 us; error 29 us; skew 500 ppm
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:39.580268 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:15:39.580694 1716 webserver.cc:460] Webserver started at http://127.0.98.2:38007/ using document root <none> and password file <none>
I20260621 05:15:39.580925 1716 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:39.580989 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:39.581116 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:39.582108 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "abfe33faa73c4e548533f20362e9e3f4"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "35b523c9b8eb21cb3058038c098e4184"
server_key_iv: "e111abca5e37932e60837d91b4139461"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.582489 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "abfe33faa73c4e548533f20362e9e3f4"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "35b523c9b8eb21cb3058038c098e4184"
server_key_iv: "e111abca5e37932e60837d91b4139461"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.583810 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:15:39.584483 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.584649 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:39.584743 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "abfe33faa73c4e548533f20362e9e3f4"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "35b523c9b8eb21cb3058038c098e4184"
server_key_iv: "e111abca5e37932e60837d91b4139461"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.584821 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:39.596762 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:39.598340 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:39.598489 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:39.598964 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:39.599025 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.599066 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:39.599090 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:39.606379 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36777
I20260621 05:15:39.606765 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36777 every 8 connection(s)
I20260621 05:15:39.606769 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:15:39.608701 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.599469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47595 (local address 127.0.98.62:45145)
0621 05:15:39.599527 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.599529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.599793 (+ 264us) server_negotiation.cc:407] Connection header received
0621 05:15:39.600306 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.600308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.600319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.600365 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:39.601091 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.602000 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.603053 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.603253 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.603344 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.605950 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:39.605952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:39.605952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:39.605958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:39.607536 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.607958 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.607959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.607959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.608007 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.608220 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.608220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.608221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.608317 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:39.608366 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.608584 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.608641 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:15:39.608865 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.599467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.2:47595)
0621 05:15:39.599658 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.599699 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.600209 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.600424 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.600427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.600763 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:39.600933 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.600938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.602099 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.602100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.602962 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.602964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.603353 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.603354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.603475 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.603995 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:39.604004 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:39.605876 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:39.607606 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.607610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.607611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.607893 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.608068 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.608068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.608068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.608156 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.608370 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:39.608373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:39.608475 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.608547 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.608654 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260621 05:15:39.609318 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45145
I20260621 05:15:39.609421 1847 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:39.609592 1847 heartbeater.cc:511] Master 127.0.98.62:45145 requested a full tablet report, sending...
I20260621 05:15:39.609952 1504 ts_manager.cc:194] Registered new tserver with Master: abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777)
I20260621 05:15:39.610436 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47595
I20260621 05:15:39.610513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 1716
I20260621 05:15:39.610648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:39.610858 392 external_mini_cluster.cc:1428] Setting key 1f9f09e392c10be11a7229a623a46bae
I20260621 05:15:39.616420 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.610338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36485 (local address 127.0.98.62:45145)
0621 05:15:39.610373 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.610374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.611113 (+ 739us) server_negotiation.cc:407] Connection header received
0621 05:15:39.611193 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.611193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.611204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.611259 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:39.611596 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.612072 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.612784 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.612977 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.613058 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.613913 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:39.613914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:39.613914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:39.613921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:39.615265 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.615620 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.615621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.615622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.615691 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.615880 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.615881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.615882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.616014 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:39.616116 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.616267 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.616333 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:15:39.616436 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.610246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.2:36485)
0621 05:15:39.611040 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.611050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.611126 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.611262 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.611263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.611461 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:39.611521 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.611522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.612164 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.612165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.612682 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.612683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.613060 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.613061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.613126 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.613566 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:39.613570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:39.613828 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:39.615365 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.615366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.615367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.615531 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.615756 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.615757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.615757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.615811 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.616109 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:39.616110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:39.616152 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.616240 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.616333 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:15:39.633544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:39.710862 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:39.711071 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:39.711107 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:39.712230 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:39.712293 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:39.712325 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:15:39.712363 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:39.713739 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:39.713975 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:39.714200 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:39.718888 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:39.718897 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:39.718930 1856 server_base.cc:1034] running on GCE node
W20260621 05:15:39.718904 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:39.719293 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:39.719511 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:39.720652 1856 hybrid_clock.cc:648] HybridClock initialized: now 1782018939720634 us; error 33 us; skew 500 ppm
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:39.723006 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:15:39.723425 1856 webserver.cc:460] Webserver started at http://127.0.98.3:36299/ using document root <none> and password file <none>
I20260621 05:15:39.723673 1856 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:39.723738 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:39.723871 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:39.724825 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b8f864534cb45858b5f0892e6075a61"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "e6a4d91281a3bce143c354824bd24767"
server_key_iv: "38fd38b5ff4fdcbf347a6ed41e43b9a7"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.725167 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b8f864534cb45858b5f0892e6075a61"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "e6a4d91281a3bce143c354824bd24767"
server_key_iv: "38fd38b5ff4fdcbf347a6ed41e43b9a7"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.726421 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:15:39.727066 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.727243 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:39.727336 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "0b8f864534cb45858b5f0892e6075a61"
format_stamp: "Formatted at 2026-06-21 05:15:39 on dist-test-slave-czm5"
server_key: "e6a4d91281a3bce143c354824bd24767"
server_key_iv: "38fd38b5ff4fdcbf347a6ed41e43b9a7"
server_key_version: "encryptionkey@0"
I20260621 05:15:39.727385 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:39.748004 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:39.749612 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:39.749712 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:39.750113 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:39.750190 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.750229 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:39.750252 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.756942 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35675
I20260621 05:15:39.756990 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35675 every 8 connection(s)
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:39.757355 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:15:39.757604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 1856
I20260621 05:15:39.757697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:39.757946 392 external_mini_cluster.cc:1428] Setting key cc8ef338ab8996cb69e97ea861f86d4d
I20260621 05:15:39.760495 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.750600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48585 (local address 127.0.98.62:45145)
0621 05:15:39.750672 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.750675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.750893 (+ 218us) server_negotiation.cc:407] Connection header received
0621 05:15:39.751367 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.751368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.751380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.751430 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:39.752338 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.753108 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.754754 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.755055 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.755112 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.757867 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:39.757868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:39.757869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:39.757876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:39.759372 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.759797 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.759798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.759798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.759840 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.760010 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.760010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.760011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.760094 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:39.760167 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.760384 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.760438 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:15:39.760669 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.750568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.3:48585)
0621 05:15:39.750801 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.750825 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.751278 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.751555 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.751559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.751970 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:39.752241 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.752247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.753202 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.753204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.754657 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.754660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.755135 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.755136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.755296 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.755880 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:39.755889 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:39.757768 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:39.759437 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.759454 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.759455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.759743 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.759876 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.759876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.759876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.759974 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.760178 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:39.760182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:39.760282 (+ 100us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.760354 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.760459 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
I20260621 05:15:39.761142 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45145
I20260621 05:15:39.761250 1987 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:39.761404 1987 heartbeater.cc:511] Master 127.0.98.62:45145 requested a full tablet report, sending...
I20260621 05:15:39.761715 1504 ts_manager.cc:194] Registered new tserver with Master: 0b8f864534cb45858b5f0892e6075a61 (127.0.98.3:35675)
I20260621 05:15:39.762136 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48585
I20260621 05:15:39.766816 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.762038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37673 (local address 127.0.98.62:45145)
0621 05:15:39.762069 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.762071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.762156 (+ 85us) server_negotiation.cc:407] Connection header received
0621 05:15:39.762179 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.762180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.762189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.762245 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:39.762581 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.763019 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.763716 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.763890 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.763943 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.764670 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:39.764671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:39.764671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:39.764676 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:39.766012 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.766285 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.766286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.766286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.766331 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.766500 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.766501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.766501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.766577 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:39.766618 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.766712 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.766765 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:15:39.766826 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.761959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.3:37673)
0621 05:15:39.762091 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.762100 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.762149 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.762260 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.762261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.762445 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:39.762499 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.762500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.763106 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.763107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.763598 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.763600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.763953 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.763953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.764000 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.764318 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:39.764336 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:39.764601 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:39.766076 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.766077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.766077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.766226 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.766404 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:39.766405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:39.766405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:39.766466 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:39.766623 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:39.766623 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:39.766653 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.766699 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.766762 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260621 05:15:39.769821 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:15:39 dist-test-slave-czm5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:39.801682 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.792989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.98.62:45145)
0621 05:15:39.793045 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.793047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.793055 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:15:39.793083 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.793084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.793099 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.793149 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:39.793964 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.794710 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.796106 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.796286 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.796346 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.799256 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:39.799257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:39.799258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:39.799264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:39.800760 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.801054 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.801055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.801055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.801129 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:39.801311 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:39.801312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:39.801312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:39.801408 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:39.801481 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.801559 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.801617 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:15:39.803376 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:15:39.805143 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:39.811336 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.807308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36777 (local address 127.0.0.1:51534)
0621 05:15:39.807590 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.807673 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.807756 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.808044 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.808047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.808049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:39.808108 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.808110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.808952 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.808953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.810173 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.810175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.811021 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.811022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.811093 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.811105 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.811186 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.811235 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":56,"threads_started":1}
I20260621 05:15:39.811357 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.807415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.0.98.2:36777)
0621 05:15:39.807587 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.807592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.807733 (+ 141us) server_negotiation.cc:407] Connection header received
0621 05:15:39.807872 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.807875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.807890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.807963 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:39.808187 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.808875 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.810242 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.810931 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.811011 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.811080 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.811199 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.811238 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260621 05:15:39.811543 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.807511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40023 (local address 127.0.0.1:34614)
0621 05:15:39.807812 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.807822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.807873 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.808143 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.808143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.808145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:39.808194 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.808195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.809496 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.809497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.810469 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.810472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.811340 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.811341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.811404 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.811408 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.811441 (+ 33us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.811479 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":7,"thread_start_us":70,"threads_started":1}
I20260621 05:15:39.811692 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.807698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35675 (local address 127.0.0.1:57906)
0621 05:15:39.807907 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.807917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.807958 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.808269 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.808270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.808272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:39.808308 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.808309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.809486 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.809486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.810586 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.810588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.811435 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.811437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.811499 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.811503 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.811554 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.811590 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":29,"thread_start_us":69,"threads_started":1}
I20260621 05:15:39.811699 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.807893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.0.98.3:35675)
0621 05:15:39.808090 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.808094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.808102 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:15:39.808134 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.808136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.808151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.808216 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:39.808390 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.809410 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.810664 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.811268 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.811327 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.811400 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.811569 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.811599 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260621 05:15:39.812058 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb010e16644ede937bf1b4235cf399 (DEFAULT_TABLE table=test-table [id=c184006e1a2b40b1bd96e68eeaae45d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:39.812584 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.807737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.0.98.1:40023)
0621 05:15:39.807993 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.807997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.808006 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:15:39.808040 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.808042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.808055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.808266 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:39.808317 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.809368 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.810583 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.811247 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.812289 (+ 1042us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.812371 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.812397 (+ 26us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.812485 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260621 05:15:39.812608 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb010e16644ede937bf1b4235cf399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:39.812669 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb010e16644ede937bf1b4235cf399 (DEFAULT_TABLE table=test-table [id=c184006e1a2b40b1bd96e68eeaae45d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:39.812908 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb010e16644ede937bf1b4235cf399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:39.813299 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb010e16644ede937bf1b4235cf399 (DEFAULT_TABLE table=test-table [id=c184006e1a2b40b1bd96e68eeaae45d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:39.813580 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb010e16644ede937bf1b4235cf399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:39.815500 2010 tablet_bootstrap.cc:492] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: Bootstrap starting.
I20260621 05:15:39.816056 2010 tablet_bootstrap.cc:654] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:39.816360 2010 log.cc:824] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:39.817073 2010 tablet_bootstrap.cc:492] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: No bootstrap required, opened a new log
I20260621 05:15:39.817147 2010 ts_tablet_manager.cc:1397] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:39.817068 2012 tablet_bootstrap.cc:492] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Bootstrap starting.
I20260621 05:15:39.817310 2013 tablet_bootstrap.cc:492] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Bootstrap starting.
I20260621 05:15:39.817760 2012 tablet_bootstrap.cc:654] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:39.817901 2013 tablet_bootstrap.cc:654] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:39.818100 2012 log.cc:824] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:39.818204 2013 log.cc:824] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:39.818836 2013 tablet_bootstrap.cc:492] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: No bootstrap required, opened a new log
I20260621 05:15:39.818938 2013 ts_tablet_manager.cc:1397] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:39.819018 2012 tablet_bootstrap.cc:492] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: No bootstrap required, opened a new log
I20260621 05:15:39.819093 2012 ts_tablet_manager.cc:1397] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:39.819345 2010 raft_consensus.cc:348] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.819514 2010 raft_consensus.cc:374] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:39.819558 2010 raft_consensus.cc:729] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8f864534cb45858b5f0892e6075a61, State: Initialized, Role: FOLLOWER
I20260621 05:15:39.819703 2010 consensus_queue.cc:260] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.819964 2010 ts_tablet_manager.cc:1428] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:15:39.820000 1987 heartbeater.cc:503] Master 127.0.98.62:45145 was elected leader, sending a full tablet report...
I20260621 05:15:39.820899 2013 raft_consensus.cc:348] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.821064 2013 raft_consensus.cc:374] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:39.821112 2013 raft_consensus.cc:729] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52da5ece527b420bad61cf43836b8cea, State: Initialized, Role: FOLLOWER
I20260621 05:15:39.821003 2012 raft_consensus.cc:348] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.821167 2012 raft_consensus.cc:374] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:39.821208 2012 raft_consensus.cc:729] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfe33faa73c4e548533f20362e9e3f4, State: Initialized, Role: FOLLOWER
I20260621 05:15:39.821223 2013 consensus_queue.cc:260] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.821313 2012 consensus_queue.cc:260] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.821472 2013 ts_tablet_manager.cc:1428] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:15:39.821520 1707 heartbeater.cc:503] Master 127.0.98.62:45145 was elected leader, sending a full tablet report...
I20260621 05:15:39.821568 2012 ts_tablet_manager.cc:1428] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:15:39.821581 1847 heartbeater.cc:503] Master 127.0.98.62:45145 was elected leader, sending a full tablet report...
W20260621 05:15:39.857520 1848 tablet.cc:2369] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:15:39.915802 2016 raft_consensus.cc:482] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:39.915910 2016 raft_consensus.cc:504] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.916251 2016 leader_election.cc:288] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777), 52da5ece527b420bad61cf43836b8cea (127.0.98.1:40023)
I20260621 05:15:39.919816 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.916532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49373 (local address 127.0.98.2:36777)
0621 05:15:39.916600 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.916603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.916613 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:15:39.916641 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.916642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.916654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.916704 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:39.916873 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.917604 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.918959 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.919475 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.919546 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.919555 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.919715 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.919749 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:15:39.919816 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.916655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34031 (local address 127.0.98.1:40023)
0621 05:15:39.916710 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.916712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.916723 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:15:39.916800 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.916801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.916810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.916884 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:39.917017 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.917651 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.918959 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.919475 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.919546 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.919555 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.919715 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.919749 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:15:39.919837 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.916434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36777 (local address 127.0.98.3:49373)
0621 05:15:39.916498 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.916509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.916585 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.916731 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.916732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.916734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:39.916803 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.916805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.917672 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.917673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.918877 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.918879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.919558 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.919558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.919654 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.919658 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.919711 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.919750 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":46}
I20260621 05:15:39.919853 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.916544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40023 (local address 127.0.98.3:34031)
0621 05:15:39.916687 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:39.916695 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:39.916729 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:39.916905 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:39.916906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:39.916907 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:39.916957 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.916958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.917716 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.917716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.918877 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:39.918879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.919558 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.919558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:39.919654 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.919658 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:39.919711 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:15:39.919753 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":35,"thread_start_us":104,"threads_started":1}
I20260621 05:15:39.920271 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb010e16644ede937bf1b4235cf399" candidate_uuid: "0b8f864534cb45858b5f0892e6075a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52da5ece527b420bad61cf43836b8cea" is_pre_election: true
I20260621 05:15:39.920331 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb010e16644ede937bf1b4235cf399" candidate_uuid: "0b8f864534cb45858b5f0892e6075a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfe33faa73c4e548533f20362e9e3f4" is_pre_election: true
I20260621 05:15:39.920423 1661 raft_consensus.cc:2454] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8f864534cb45858b5f0892e6075a61 in term 0.
I20260621 05:15:39.920437 1801 raft_consensus.cc:2454] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8f864534cb45858b5f0892e6075a61 in term 0.
I20260621 05:15:39.920624 1875 leader_election.cc:302] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8f864534cb45858b5f0892e6075a61, abfe33faa73c4e548533f20362e9e3f4; no voters:
I20260621 05:15:39.920750 2016 raft_consensus.cc:2781] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:39.920828 2016 raft_consensus.cc:482] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:39.920861 2016 raft_consensus.cc:3037] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:39.921818 2016 raft_consensus.cc:504] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.921936 2016 leader_election.cc:288] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [CANDIDATE]: Term 1 election: Requested vote from peers abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777), 52da5ece527b420bad61cf43836b8cea (127.0.98.1:40023)
I20260621 05:15:39.922053 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb010e16644ede937bf1b4235cf399" candidate_uuid: "0b8f864534cb45858b5f0892e6075a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfe33faa73c4e548533f20362e9e3f4"
I20260621 05:15:39.922111 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb010e16644ede937bf1b4235cf399" candidate_uuid: "0b8f864534cb45858b5f0892e6075a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52da5ece527b420bad61cf43836b8cea"
I20260621 05:15:39.922139 1801 raft_consensus.cc:3037] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:39.922180 1661 raft_consensus.cc:3037] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:39.922966 1661 raft_consensus.cc:2454] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8f864534cb45858b5f0892e6075a61 in term 1.
I20260621 05:15:39.922966 1801 raft_consensus.cc:2454] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8f864534cb45858b5f0892e6075a61 in term 1.
I20260621 05:15:39.923133 1875 leader_election.cc:302] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8f864534cb45858b5f0892e6075a61, abfe33faa73c4e548533f20362e9e3f4; no voters:
I20260621 05:15:39.923213 2016 raft_consensus.cc:2781] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:39.923357 2016 raft_consensus.cc:686] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 1 LEADER]: Becoming Leader. State: Replica: 0b8f864534cb45858b5f0892e6075a61, State: Running, Role: LEADER
I20260621 05:15:39.923472 2016 consensus_queue.cc:237] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.924253 1503 catalog_manager.cc:5495] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b8f864534cb45858b5f0892e6075a61 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260621 05:15:39.954959 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:39.952758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.0.98.3:35675)
0621 05:15:39.952815 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:39.952816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:39.952844 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:15:39.952927 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:39.952928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:39.952944 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:39.952997 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:39.953121 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.953580 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.954293 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:39.954466 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:39.954516 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:39.954668 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:39.954752 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:39.954806 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:39.954850 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 05:15:39.954864 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:15:39.956696 1661 raft_consensus.cc:1261] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Refusing update from remote peer 0b8f864534cb45858b5f0892e6075a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:39.956740 1801 raft_consensus.cc:1261] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Refusing update from remote peer 0b8f864534cb45858b5f0892e6075a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:39.957024 2020 consensus_queue.cc:1035] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:15:39.957098 1708 tablet.cc:2369] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:15:39.957134 2016 consensus_queue.cc:1035] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:39.961254 2025 mvcc.cc:205] Tried to move back new op lower bound from 7299149578060005376 to 7299149577927262208. Current Snapshot: MvccSnapshot[applied={T|T < 7299149578060005376 or (T in {7299149578060005376})}]
I20260621 05:15:39.962744 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:39.962814 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:39.963490 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217: Sending DeleteTablet for 3 replicas of tablet 9ecb010e16644ede937bf1b4235cf399
I20260621 05:15:39.963819 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb010e16644ede937bf1b4235cf399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34614
I20260621 05:15:39.963904 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb010e16644ede937bf1b4235cf399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51534
I20260621 05:15:39.964023 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb010e16644ede937bf1b4235cf399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57906
I20260621 05:15:39.964071 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 05:15:39.964198 2032 tablet_replica.cc:330] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: stopping tablet replica
W20260621 05:15:39.964210 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:39.964277 2032 raft_consensus.cc:2229] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:39.964370 2032 raft_consensus.cc:2258] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:39.964375 2034 tablet_replica.cc:330] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: stopping tablet replica
I20260621 05:15:39.964560 2034 raft_consensus.cc:2229] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:15:39.964680 2034 raft_consensus.cc:2258] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:39.964939 2032 ts_tablet_manager.cc:1905] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:39.965148 2034 ts_tablet_manager.cc:1905] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:39.965636 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 (DEFAULT_TABLE table=test-table [id=6a6736210f3d4e52bae3d0b11f8f47fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:39.965689 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 (DEFAULT_TABLE table=test-table [id=6a6736210f3d4e52bae3d0b11f8f47fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:39.965767 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:39.965783 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:39.966924 2013 tablet_bootstrap.cc:492] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: Bootstrap starting.
I20260621 05:15:39.966984 2010 tablet_bootstrap.cc:492] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: Bootstrap starting.
I20260621 05:15:39.967119 2034 ts_tablet_manager.cc:1918] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:15:39.967223 2034 log.cc:1196] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/9ecb010e16644ede937bf1b4235cf399
I20260621 05:15:39.967438 2010 tablet_bootstrap.cc:654] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:39.967590 2034 ts_tablet_manager.cc:1939] T 9ecb010e16644ede937bf1b4235cf399 P 0b8f864534cb45858b5f0892e6075a61: Deleting consensus metadata
I20260621 05:15:39.967607 2013 tablet_bootstrap.cc:654] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:39.967960 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 (DEFAULT_TABLE table=test-table [id=6a6736210f3d4e52bae3d0b11f8f47fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:39.968071 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:39.968918 2032 ts_tablet_manager.cc:1918] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:15:39.968992 2032 log.cc:1196] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/9ecb010e16644ede937bf1b4235cf399
I20260621 05:15:39.969221 2032 ts_tablet_manager.cc:1939] T 9ecb010e16644ede937bf1b4235cf399 P 52da5ece527b420bad61cf43836b8cea: Deleting consensus metadata
I20260621 05:15:39.969624 2013 tablet_bootstrap.cc:492] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: No bootstrap required, opened a new log
I20260621 05:15:39.969687 2013 ts_tablet_manager.cc:1397] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260621 05:15:39.969676 1491 catalog_manager.cc:4841] TS 0b8f864534cb45858b5f0892e6075a61 (127.0.98.3:35675): tablet 9ecb010e16644ede937bf1b4235cf399 (table test-table [id=c184006e1a2b40b1bd96e68eeaae45d1]) successfully deleted
I20260621 05:15:39.969766 1491 catalog_manager.cc:4841] TS 52da5ece527b420bad61cf43836b8cea (127.0.98.1:40023): tablet 9ecb010e16644ede937bf1b4235cf399 (table test-table [id=c184006e1a2b40b1bd96e68eeaae45d1]) successfully deleted
I20260621 05:15:39.969812 2013 raft_consensus.cc:348] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.969887 2013 raft_consensus.cc:374] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:39.969913 2013 raft_consensus.cc:729] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52da5ece527b420bad61cf43836b8cea, State: Initialized, Role: FOLLOWER
I20260621 05:15:39.969959 2013 consensus_queue.cc:260] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.970057 2013 ts_tablet_manager.cc:1428] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.970134 2010 tablet_bootstrap.cc:492] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: No bootstrap required, opened a new log
I20260621 05:15:39.970146 2012 tablet_bootstrap.cc:492] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: Bootstrap starting.
I20260621 05:15:39.970180 2010 ts_tablet_manager.cc:1397] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260621 05:15:39.970315 2010 raft_consensus.cc:348] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.970383 2010 raft_consensus.cc:374] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:39.970403 2010 raft_consensus.cc:729] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8f864534cb45858b5f0892e6075a61, State: Initialized, Role: FOLLOWER
I20260621 05:15:39.970438 2010 consensus_queue.cc:260] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.970537 2010 ts_tablet_manager.cc:1428] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.970541 2012 tablet_bootstrap.cc:654] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:39.970971 2033 tablet_replica.cc:330] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: stopping tablet replica
I20260621 05:15:39.971086 2033 raft_consensus.cc:2229] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:39.971215 2033 raft_consensus.cc:2258] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:39.971635 2012 tablet_bootstrap.cc:492] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: No bootstrap required, opened a new log
I20260621 05:15:39.971690 2012 ts_tablet_manager.cc:1397] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:39.971789 2033 ts_tablet_manager.cc:1905] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:39.971827 2012 raft_consensus.cc:348] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.971903 2012 raft_consensus.cc:374] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:39.971935 2012 raft_consensus.cc:729] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfe33faa73c4e548533f20362e9e3f4, State: Initialized, Role: FOLLOWER
I20260621 05:15:39.971985 2012 consensus_queue.cc:260] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:39.972077 2012 ts_tablet_manager.cc:1428] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:39.973568 2033 ts_tablet_manager.cc:1918] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:15:39.973623 2033 log.cc:1196] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/9ecb010e16644ede937bf1b4235cf399
I20260621 05:15:39.973862 2033 ts_tablet_manager.cc:1939] T 9ecb010e16644ede937bf1b4235cf399 P abfe33faa73c4e548533f20362e9e3f4: Deleting consensus metadata
I20260621 05:15:39.974290 1491 catalog_manager.cc:4841] TS abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777): tablet 9ecb010e16644ede937bf1b4235cf399 (table test-table [id=c184006e1a2b40b1bd96e68eeaae45d1]) successfully deleted
W20260621 05:15:40.008438 1988 tablet.cc:2369] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:15:40.069501 2017 raft_consensus.cc:482] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:40.069597 2017 raft_consensus.cc:504] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.069949 2017 leader_election.cc:288] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777), 0b8f864534cb45858b5f0892e6075a61 (127.0.98.3:35675)
I20260621 05:15:40.073467 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.070264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53407 (local address 127.0.98.2:36777)
0621 05:15:40.070332 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.070334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.070425 (+ 91us) server_negotiation.cc:407] Connection header received
0621 05:15:40.070521 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.070522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.070531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.070598 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:40.070768 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.071505 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.072747 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.073106 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.073188 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.073200 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.073365 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.073402 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:15:40.073480 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.070164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36777 (local address 127.0.98.1:53407)
0621 05:15:40.070356 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:40.070367 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:40.070456 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:40.070609 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:40.070610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:40.070612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:40.070692 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.070694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.071587 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.071588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.072648 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.072650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.073201 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.073202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.073288 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.073292 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:40.073360 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:15:40.073399 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260621 05:15:40.073539 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.070426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58335 (local address 127.0.98.3:35675)
0621 05:15:40.070464 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.070466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.070506 (+ 40us) server_negotiation.cc:407] Connection header received
0621 05:15:40.070586 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.070587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.070596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.070642 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:40.070798 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.071270 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.072711 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.073176 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.073244 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.073254 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.073447 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.073478 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:15:40.073665 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2459b555ea3a4a7fb48cfd0cd6dd74d0" candidate_uuid: "52da5ece527b420bad61cf43836b8cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfe33faa73c4e548533f20362e9e3f4" is_pre_election: true
I20260621 05:15:40.073760 1801 raft_consensus.cc:2454] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52da5ece527b420bad61cf43836b8cea in term 0.
I20260621 05:15:40.073766 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.070366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35675 (local address 127.0.98.1:58335)
0621 05:15:40.070454 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:40.070463 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:40.070529 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:40.070680 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:40.070680 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:40.070681 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:40.070742 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.070744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.071371 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.071372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.072609 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.072612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.073308 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.073310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.073374 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.073378 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:40.073644 (+ 266us) client_negotiation.cc:240] Negotiation successful
0621 05:15:40.073684 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":48,"threads_started":1}
I20260621 05:15:40.073951 1595 leader_election.cc:302] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52da5ece527b420bad61cf43836b8cea, abfe33faa73c4e548533f20362e9e3f4; no voters:
I20260621 05:15:40.074003 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2459b555ea3a4a7fb48cfd0cd6dd74d0" candidate_uuid: "52da5ece527b420bad61cf43836b8cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8f864534cb45858b5f0892e6075a61" is_pre_election: true
I20260621 05:15:40.074067 2017 raft_consensus.cc:2781] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:40.074095 2017 raft_consensus.cc:482] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:40.074163 2017 raft_consensus.cc:3037] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:40.074143 1941 raft_consensus.cc:2454] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52da5ece527b420bad61cf43836b8cea in term 0.
I20260621 05:15:40.074874 2017 raft_consensus.cc:504] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.075009 2017 leader_election.cc:288] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [CANDIDATE]: Term 1 election: Requested vote from peers abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777), 0b8f864534cb45858b5f0892e6075a61 (127.0.98.3:35675)
I20260621 05:15:40.075134 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2459b555ea3a4a7fb48cfd0cd6dd74d0" candidate_uuid: "52da5ece527b420bad61cf43836b8cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfe33faa73c4e548533f20362e9e3f4"
I20260621 05:15:40.075167 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2459b555ea3a4a7fb48cfd0cd6dd74d0" candidate_uuid: "52da5ece527b420bad61cf43836b8cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8f864534cb45858b5f0892e6075a61"
I20260621 05:15:40.075207 1801 raft_consensus.cc:3037] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:40.075223 1941 raft_consensus.cc:3037] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:40.075953 1801 raft_consensus.cc:2454] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52da5ece527b420bad61cf43836b8cea in term 1.
I20260621 05:15:40.075951 1941 raft_consensus.cc:2454] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52da5ece527b420bad61cf43836b8cea in term 1.
I20260621 05:15:40.076135 1595 leader_election.cc:302] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8f864534cb45858b5f0892e6075a61, 52da5ece527b420bad61cf43836b8cea; no voters:
I20260621 05:15:40.076252 2017 raft_consensus.cc:2781] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:40.076370 2017 raft_consensus.cc:686] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 1 LEADER]: Becoming Leader. State: Replica: 52da5ece527b420bad61cf43836b8cea, State: Running, Role: LEADER
I20260621 05:15:40.076453 2017 consensus_queue.cc:237] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.077242 1504 catalog_manager.cc:5495] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea reported cstate change: term changed from 0 to 1, leader changed from <none> to 52da5ece527b420bad61cf43836b8cea (127.0.98.1). New cstate: current_term: 1 leader_uuid: "52da5ece527b420bad61cf43836b8cea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } health_report { overall_health: UNKNOWN } } }
Jun 21 05:15:40 dist-test-slave-czm5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:40.114622 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.108367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.0.98.62:45145)
0621 05:15:40.108442 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.108444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.108540 (+ 96us) server_negotiation.cc:407] Connection header received
0621 05:15:40.108612 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.108613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.108622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.108686 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:40.109175 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.109608 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.110299 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.110457 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.110538 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.112397 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:40.112398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:40.112399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:40.112404 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:40.113722 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:40.114018 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:40.114018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:40.114018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:40.114063 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:40.114243 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:40.114244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:40.114244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:40.114344 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:40.114391 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.114500 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.114560 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:15:40.114771 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.108297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.0.1:34962)
0621 05:15:40.108466 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:40.108478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:40.108568 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:40.108690 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:40.108691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:40.109056 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:40.109107 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.109109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.109682 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.109683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.110215 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.110217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.110538 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.110538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.110583 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.111001 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:40.111004 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:40.112307 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:40.113791 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:40.113795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:40.113797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:40.113958 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:40.114126 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:40.114126 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:40.114126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:40.114194 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:40.114394 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:40.114398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:40.114435 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 05:15:40.114486 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:15:40.114623 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":56,"thread_start_us":94,"threads_started":1}
I20260621 05:15:40.116556 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34962:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:15:40.116737 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:40.117929 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 90eb6994418e448a8a2b3ce209eda44f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b3cc329f6dd4302bbda33bcf25483a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:15:40.117962 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 90eb6994418e448a8a2b3ce209eda44f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b3cc329f6dd4302bbda33bcf25483a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:15:40.118101 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90eb6994418e448a8a2b3ce209eda44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:40.118100 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90eb6994418e448a8a2b3ce209eda44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:40.119112 2010 tablet_bootstrap.cc:492] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: Bootstrap starting.
I20260621 05:15:40.119211 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 90eb6994418e448a8a2b3ce209eda44f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b3cc329f6dd4302bbda33bcf25483a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:15:40.119349 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90eb6994418e448a8a2b3ce209eda44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:40.119447 2010 tablet_bootstrap.cc:654] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:40.119716 2013 tablet_bootstrap.cc:492] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: Bootstrap starting.
I20260621 05:15:40.120096 2013 tablet_bootstrap.cc:654] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:40.121054 2010 tablet_bootstrap.cc:492] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: No bootstrap required, opened a new log
I20260621 05:15:40.121101 2010 ts_tablet_manager.cc:1397] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:15:40.121196 2010 raft_consensus.cc:348] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.121256 2010 raft_consensus.cc:374] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:40.121279 2010 raft_consensus.cc:729] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8f864534cb45858b5f0892e6075a61, State: Initialized, Role: FOLLOWER
I20260621 05:15:40.121330 2010 consensus_queue.cc:260] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.121425 2010 ts_tablet_manager.cc:1428] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:40.121443 2020 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.121497 2012 tablet_bootstrap.cc:492] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: Bootstrap starting.
I20260621 05:15:40.121505 2020 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:40.121793 2012 tablet_bootstrap.cc:654] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:40.122212 2013 tablet_bootstrap.cc:492] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: No bootstrap required, opened a new log
I20260621 05:15:40.122263 2013 ts_tablet_manager.cc:1397] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260621 05:15:40.122413 2013 raft_consensus.cc:348] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.122466 2013 raft_consensus.cc:374] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:40.122478 2013 raft_consensus.cc:729] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52da5ece527b420bad61cf43836b8cea, State: Initialized, Role: FOLLOWER
I20260621 05:15:40.122530 2013 consensus_queue.cc:260] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.122619 2013 ts_tablet_manager.cc:1428] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:40.122660 2017 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.122741 2017 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:40.122809 2012 tablet_bootstrap.cc:492] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: No bootstrap required, opened a new log
I20260621 05:15:40.122862 2012 ts_tablet_manager.cc:1397] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:40.123000 2012 raft_consensus.cc:348] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.123071 2012 raft_consensus.cc:374] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:40.123093 2012 raft_consensus.cc:729] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfe33faa73c4e548533f20362e9e3f4, State: Initialized, Role: FOLLOWER
I20260621 05:15:40.123143 2012 consensus_queue.cc:260] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.123232 2012 ts_tablet_manager.cc:1428] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:40.123349 2018 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.123456 2018 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:40.133811 2020 raft_consensus.cc:482] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:40.133919 2020 raft_consensus.cc:504] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.134063 2020 leader_election.cc:288] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52da5ece527b420bad61cf43836b8cea (127.0.98.1:40023), abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777)
I20260621 05:15:40.134326 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90eb6994418e448a8a2b3ce209eda44f" candidate_uuid: "0b8f864534cb45858b5f0892e6075a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfe33faa73c4e548533f20362e9e3f4" is_pre_election: true
I20260621 05:15:40.134332 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90eb6994418e448a8a2b3ce209eda44f" candidate_uuid: "0b8f864534cb45858b5f0892e6075a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52da5ece527b420bad61cf43836b8cea" is_pre_election: true
I20260621 05:15:40.134438 1801 raft_consensus.cc:2454] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8f864534cb45858b5f0892e6075a61 in term 0.
I20260621 05:15:40.134438 1661 raft_consensus.cc:2454] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8f864534cb45858b5f0892e6075a61 in term 0.
I20260621 05:15:40.134590 1875 leader_election.cc:302] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8f864534cb45858b5f0892e6075a61, 52da5ece527b420bad61cf43836b8cea; no voters:
I20260621 05:15:40.134684 2020 raft_consensus.cc:2781] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:40.134745 2020 raft_consensus.cc:482] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:40.134785 2020 raft_consensus.cc:3037] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:40.135785 2020 raft_consensus.cc:504] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.135912 2020 leader_election.cc:288] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [CANDIDATE]: Term 1 election: Requested vote from peers 52da5ece527b420bad61cf43836b8cea (127.0.98.1:40023), abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777)
I20260621 05:15:40.136198 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90eb6994418e448a8a2b3ce209eda44f" candidate_uuid: "0b8f864534cb45858b5f0892e6075a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52da5ece527b420bad61cf43836b8cea"
I20260621 05:15:40.136189 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90eb6994418e448a8a2b3ce209eda44f" candidate_uuid: "0b8f864534cb45858b5f0892e6075a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfe33faa73c4e548533f20362e9e3f4"
I20260621 05:15:40.136269 1801 raft_consensus.cc:3037] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:40.136270 1661 raft_consensus.cc:3037] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:40.136961 1801 raft_consensus.cc:2454] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8f864534cb45858b5f0892e6075a61 in term 1.
I20260621 05:15:40.137004 1661 raft_consensus.cc:2454] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8f864534cb45858b5f0892e6075a61 in term 1.
I20260621 05:15:40.137138 1875 leader_election.cc:302] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8f864534cb45858b5f0892e6075a61, 52da5ece527b420bad61cf43836b8cea; no voters:
I20260621 05:15:40.137220 2020 raft_consensus.cc:2781] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:40.137306 2020 raft_consensus.cc:686] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [term 1 LEADER]: Becoming Leader. State: Replica: 0b8f864534cb45858b5f0892e6075a61, State: Running, Role: LEADER
I20260621 05:15:40.137370 2020 consensus_queue.cc:237] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } }
I20260621 05:15:40.137666 2016 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: TxnStatusTablet state changed. Reason: New leader 0b8f864534cb45858b5f0892e6075a61. Latest consensus state: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.137753 2016 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:15:40.137952 1503 catalog_manager.cc:5495] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b8f864534cb45858b5f0892e6075a61 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260621 05:15:40.138253 2052 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:15:40.138386 2052 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:15:40.146186 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.141000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35675 (local address 127.0.0.1:57924)
0621 05:15:40.141069 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:40.141079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:40.141124 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:40.143868 (+ 2744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:40.143869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:40.143870 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:15:40.143909 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.143911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.144789 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.144790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.145342 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.145344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.145797 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.145798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.145847 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.145860 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:15:40.146049 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:15:40.146055 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:15:40.146089 (+ 34us) client_negotiation.cc:240] Negotiation successful
0621 05:15:40.146135 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:15:40.146191 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.143693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.0.98.3:35675)
0621 05:15:40.143750 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.143752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.143761 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:15:40.143782 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.143783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.143794 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.143854 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:40.143959 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.144683 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.145422 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.145717 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.145795 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.145944 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:40.145976 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:40.146044 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.146092 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.146123 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:15:40.147053 1801 raft_consensus.cc:1261] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Refusing update from remote peer 0b8f864534cb45858b5f0892e6075a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:40.147053 1661 raft_consensus.cc:1261] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Refusing update from remote peer 0b8f864534cb45858b5f0892e6075a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:40.147244 2016 consensus_queue.cc:1035] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:40.147331 2020 consensus_queue.cc:1035] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:40.147868 2018 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: TxnStatusTablet state changed. Reason: New leader 0b8f864534cb45858b5f0892e6075a61. Latest consensus state: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.147967 2018 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:40.148358 2017 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: TxnStatusTablet state changed. Reason: New leader 0b8f864534cb45858b5f0892e6075a61. Latest consensus state: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.148442 2017 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:40.148432 2020 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.148509 2020 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:15:40.148962 2023 mvcc.cc:205] Tried to move back new op lower bound from 7299149578840883200 to 7299149578802991104. Current Snapshot: MvccSnapshot[applied={T|T < 7299149578840883200 or (T in {7299149578840883200})}]
I20260621 05:15:40.149286 2018 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.149354 2018 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P abfe33faa73c4e548533f20362e9e3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:40.152168 2016 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.152236 2016 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P 0b8f864534cb45858b5f0892e6075a61: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:15:40.152351 2024 mvcc.cc:205] Tried to move back new op lower bound from 7299149578840883200 to 7299149578802991104. Current Snapshot: MvccSnapshot[applied={T|T < 7299149578840883200}]
I20260621 05:15:40.152611 2017 tablet_replica.cc:439] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b8f864534cb45858b5f0892e6075a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52da5ece527b420bad61cf43836b8cea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40023 } } peers { permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 } } peers { permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 } } }
I20260621 05:15:40.152681 2017 tablet_replica.cc:442] T 90eb6994418e448a8a2b3ce209eda44f P 52da5ece527b420bad61cf43836b8cea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:40.153280 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.150584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.0.98.1:40023)
0621 05:15:40.150627 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.150629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.150638 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:15:40.150711 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.150712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.150743 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.150821 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:40.150917 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.151359 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.152322 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.152589 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.152666 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.152895 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:40.153013 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:40.153113 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.153183 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.153208 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:15:40.158030 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.155564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39817 (local address 127.0.98.3:35675)
0621 05:15:40.155646 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.155648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.155684 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:15:40.155756 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.155757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.155767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.155820 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:40.155967 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.156410 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.157293 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.157496 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.157576 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.157776 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:40.157806 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:40.157871 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.157940 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.157957 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:15:40.158082 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.155445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35675 (local address 127.0.98.1:39817)
0621 05:15:40.155603 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:40.155614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:40.155700 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:40.155824 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:40.155824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:40.155834 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:15:40.155894 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.155896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.156488 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.156489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.157198 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.157200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.157579 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.157580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.157650 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.157667 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:15:40.157885 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:15:40.157887 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:15:40.157936 (+ 49us) client_negotiation.cc:240] Negotiation successful
0621 05:15:40.157985 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260621 05:15:40.160594 1801 raft_consensus.cc:1261] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Refusing update from remote peer 52da5ece527b420bad61cf43836b8cea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:40.160671 1941 raft_consensus.cc:1261] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Refusing update from remote peer 52da5ece527b420bad61cf43836b8cea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:40.160867 2040 consensus_queue.cc:1035] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "abfe33faa73c4e548533f20362e9e3f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:40.160987 2017 consensus_queue.cc:1035] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8f864534cb45858b5f0892e6075a61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:40.168637 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.166246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41851 (local address 127.0.98.1:40023)
0621 05:15:40.166283 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.166285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.166297 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:15:40.166370 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.166371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.166380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.166433 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:40.166510 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.167185 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.167956 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.168191 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.168270 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.168436 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:40.168465 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:40.168510 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.168564 (+ 54us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.168578 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:15:40.168663 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.166209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40023 (local address 127.0.98.3:41851)
0621 05:15:40.166267 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:40.166276 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:40.166308 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:40.166434 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:40.166435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:40.166436 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:15:40.166476 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.166476 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.167268 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.167269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.167870 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:40.167872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.168272 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.168274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:40.168325 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.168350 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:15:40.168510 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:15:40.168512 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:15:40.168560 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:15:40.168596 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260621 05:15:40.178256 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:40.178371 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:40.178891 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d194049e78ef43e0bf7f2d2be3b6e217: Sending DeleteTablet for 3 replicas of tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0
I20260621 05:15:40.179167 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51534
I20260621 05:15:40.179265 2033 tablet_replica.cc:330] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: stopping tablet replica
I20260621 05:15:40.179314 2033 raft_consensus.cc:2229] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:40.179345 2033 raft_consensus.cc:2258] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:40.179675 2033 ts_tablet_manager.cc:1905] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:40.181384 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57906
I20260621 05:15:40.181422 2033 ts_tablet_manager.cc:1918] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:40.181476 2034 tablet_replica.cc:330] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: stopping tablet replica
I20260621 05:15:40.181479 2033 log.cc:1196] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/2459b555ea3a4a7fb48cfd0cd6dd74d0
I20260621 05:15:40.181519 2034 raft_consensus.cc:2229] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:40.181552 2034 raft_consensus.cc:2258] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:40.181694 2033 ts_tablet_manager.cc:1939] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P abfe33faa73c4e548533f20362e9e3f4: Deleting consensus metadata
I20260621 05:15:40.181856 2034 ts_tablet_manager.cc:1905] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:40.183216 2034 ts_tablet_manager.cc:1918] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:40.183277 2034 log.cc:1196] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/2459b555ea3a4a7fb48cfd0cd6dd74d0
I20260621 05:15:40.183488 2034 ts_tablet_manager.cc:1939] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 0b8f864534cb45858b5f0892e6075a61: Deleting consensus metadata
I20260621 05:15:40.183854 1491 catalog_manager.cc:4841] TS abfe33faa73c4e548533f20362e9e3f4 (127.0.98.2:36777): tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 (table test-table [id=6a6736210f3d4e52bae3d0b11f8f47fc]) successfully deleted
I20260621 05:15:40.184007 1491 catalog_manager.cc:4841] TS 0b8f864534cb45858b5f0892e6075a61 (127.0.98.3:35675): tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 (table test-table [id=6a6736210f3d4e52bae3d0b11f8f47fc]) successfully deleted
I20260621 05:15:40.184171 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34614
I20260621 05:15:40.184304 2032 tablet_replica.cc:330] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: stopping tablet replica
I20260621 05:15:40.184366 2032 raft_consensus.cc:2229] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:15:40.184450 2032 raft_consensus.cc:2258] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:40.184798 2032 ts_tablet_manager.cc:1905] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:40.186295 2032 ts_tablet_manager.cc:1918] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:40.186353 2032 log.cc:1196] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/2459b555ea3a4a7fb48cfd0cd6dd74d0
I20260621 05:15:40.186602 2032 ts_tablet_manager.cc:1939] T 2459b555ea3a4a7fb48cfd0cd6dd74d0 P 52da5ece527b420bad61cf43836b8cea: Deleting consensus metadata
I20260621 05:15:40.186914 1491 catalog_manager.cc:4841] TS 52da5ece527b420bad61cf43836b8cea (127.0.98.1:40023): tablet 2459b555ea3a4a7fb48cfd0cd6dd74d0 (table test-table [id=6a6736210f3d4e52bae3d0b11f8f47fc]) successfully deleted
Jun 21 05:15:40 dist-test-slave-czm5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:15:40.201772 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.184079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.0.98.1:40023)
0621 05:15:40.184152 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.184154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.191844 (+ 7690us) server_negotiation.cc:407] Connection header received
0621 05:15:40.191900 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.191900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.191911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.191973 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:40.192629 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.193349 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.194439 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.194748 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.194820 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.198312 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:40.198319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:40.198319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:40.198327 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:40.200503 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:40.200899 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:40.200903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:40.200903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:40.200980 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:40.201201 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:40.201201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:40.201202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:40.201357 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:40.201530 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.201571 (+ 41us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.201646 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 05:15:40.202014 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34626
I20260621 05:15:40.211774 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:40.202927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.98.62:45145)
0621 05:15:40.202968 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:40.202970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:40.203139 (+ 169us) server_negotiation.cc:407] Connection header received
0621 05:15:40.203201 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:40.203202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:40.203214 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:40.203278 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:40.203904 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.204646 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.205660 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:40.205970 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:40.206038 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:40.208985 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:40.208986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:40.208986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:40.208994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:40.210584 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:40.210975 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:40.210976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:40.210976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:40.211034 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:40.211241 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:40.211254 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:40.211254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:40.211389 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:40.211481 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:40.211587 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 05:15:40.211681 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:15:40.212045 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34974
I20260621 05:15:40.212711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 1576
I20260621 05:15:40.218274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 1716
I20260621 05:15:40.223126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 1856
I20260621 05:15:40.228524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 1472
2026-06-21T05:15:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2194 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:40 dist-test-slave-czm5 krb5kdc[2073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:40 dist-test-slave-czm5 krb5kdc[2073](info): set up 2 sockets
Jun 21 05:15:40 dist-test-slave-czm5 krb5kdc[2073](info): commencing operation
krb5kdc: starting...
W20260621 05:15:41.239274 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/test-user.keytab.
W20260621 05:15:41.771471 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.454s user 0.000s sys 0.001s
Jun 21 05:15:41 dist-test-slave-czm5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260621 05:15:42.865032 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 1.074s user 0.000s sys 0.002s
I20260621 05:15:42.865430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:42.946717 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:42.946902 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:42.946923 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:42.948117 2086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:42.948156 2086 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:15:42.948199 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:42.948230 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:42.948262 2086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:42.948283 2086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:42.949759 2086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46235
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:42.949990 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:42.950217 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:42.954854 2092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:42.954890 2091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:42.954983 2086 server_base.cc:1034] running on GCE node
W20260621 05:15:42.954833 2094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:42.955267 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:42.955564 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:42.956715 2086 hybrid_clock.cc:648] HybridClock initialized: now 1782018942956698 us; error 42 us; skew 500 ppm
Jun 21 05:15:42 dist-test-slave-czm5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:42.959110 2086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:42.959556 2086 webserver.cc:460] Webserver started at http://127.0.98.62:37439/ using document root <none> and password file <none>
I20260621 05:15:42.959805 2086 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:42.959885 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:42.960029 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:42.960934 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "6ff41bd597344307baabf1dbf0927c46"
format_stamp: "Formatted at 2026-06-21 05:15:42 on dist-test-slave-czm5"
server_key: "27c87f99e00a21b180864276a61bf49e"
server_key_iv: "4763159c797a93431860191fbfa22a5c"
server_key_version: "encryptionkey@0"
I20260621 05:15:42.961287 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ff41bd597344307baabf1dbf0927c46"
format_stamp: "Formatted at 2026-06-21 05:15:42 on dist-test-slave-czm5"
server_key: "27c87f99e00a21b180864276a61bf49e"
server_key_iv: "4763159c797a93431860191fbfa22a5c"
server_key_version: "encryptionkey@0"
I20260621 05:15:42.962527 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:42.963198 2101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:42.963371 2086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:42.963471 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "6ff41bd597344307baabf1dbf0927c46"
format_stamp: "Formatted at 2026-06-21 05:15:42 on dist-test-slave-czm5"
server_key: "27c87f99e00a21b180864276a61bf49e"
server_key_iv: "4763159c797a93431860191fbfa22a5c"
server_key_version: "encryptionkey@0"
I20260621 05:15:42.963552 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:42.985899 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:42.987829 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:42.987982 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:42.992573 2086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46235
I20260621 05:15:42.992635 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46235 every 8 connection(s)
I20260621 05:15:42.992929 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:42.993518 2164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:42.995760 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: Bootstrap starting.
I20260621 05:15:42.996322 2164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:42.996610 2164 log.cc:824] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:42.997256 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: No bootstrap required, opened a new log
I20260621 05:15:42.999267 2164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff41bd597344307baabf1dbf0927c46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46235 } }
I20260621 05:15:42.999404 2164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:42.999426 2164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff41bd597344307baabf1dbf0927c46, State: Initialized, Role: FOLLOWER
I20260621 05:15:42.999599 2164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff41bd597344307baabf1dbf0927c46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46235 } }
I20260621 05:15:42.999707 2164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:42.999749 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:42.999796 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:43.000648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 2086
I20260621 05:15:43.000720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:43.000834 392 external_mini_cluster.cc:1428] Setting key 0de255b3ca200b9baaac685c8c31deb4
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:43.009405 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.001215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.0.98.62:46235)
0621 05:15:43.001355 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.001365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.001397 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:15:43.001502 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.001507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.001546 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.001674 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.002305 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.003129 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.003921 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.004311 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.004441 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.006568 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.006572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.006573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.006584 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.008000 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.008340 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.008343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.008343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.008425 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.008619 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.008620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.008621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.008843 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.008982 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.009079 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.009210 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260621 05:15:43.062783 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff41bd597344307baabf1dbf0927c46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46235 } }
I20260621 05:15:43.062973 2164 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff41bd597344307baabf1dbf0927c46; no voters:
I20260621 05:15:43.063210 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:43.063277 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:43.063541 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff41bd597344307baabf1dbf0927c46, State: Running, Role: LEADER
I20260621 05:15:43.063736 2164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:43.063706 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff41bd597344307baabf1dbf0927c46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46235 } }
I20260621 05:15:43.064217 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff41bd597344307baabf1dbf0927c46. Latest consensus state: current_term: 1 leader_uuid: "6ff41bd597344307baabf1dbf0927c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff41bd597344307baabf1dbf0927c46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46235 } } }
I20260621 05:15:43.064249 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff41bd597344307baabf1dbf0927c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff41bd597344307baabf1dbf0927c46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46235 } } }
I20260621 05:15:43.064400 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:43.064412 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:43.064927 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:43.065488 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:43.067359 2178 catalog_manager.cc:1347] Generated new cluster ID: 24d7f36194024b29884549756018fd86
I20260621 05:15:43.067425 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:43.077617 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:43.078141 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:43.085386 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: Generated new TSK 0
I20260621 05:15:43.085558 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260621 05:15:43.300236 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.213s user 0.000s sys 0.003s
I20260621 05:15:43.300592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46235
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:43.376256 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:43.376432 2190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:43.376456 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:43.377566 2190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:43.377606 2190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:43.377626 2190 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:15:43.377671 2190 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:15:43.377697 2190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:43.379091 2190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:43.379287 2190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:43.379508 2190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:43.384156 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:43.384178 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:43.384191 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:43.384380 2190 server_base.cc:1034] running on GCE node
I20260621 05:15:43.384621 2190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:43.384907 2190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:43.386080 2190 hybrid_clock.cc:648] HybridClock initialized: now 1782018943386053 us; error 39 us; skew 500 ppm
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:43.388262 2190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:43.388674 2190 webserver.cc:460] Webserver started at http://127.0.98.1:38613/ using document root <none> and password file <none>
I20260621 05:15:43.388878 2190 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:43.388942 2190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:43.389087 2190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:43.390060 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "424a3781612016a02b739636aea3fa9e"
server_key_iv: "cd4d66de0353511d3ac2291a095a7cc3"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.390415 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "424a3781612016a02b739636aea3fa9e"
server_key_iv: "cd4d66de0353511d3ac2291a095a7cc3"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.391711 2190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:43.392378 2205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.392550 2190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:43.392642 2190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "424a3781612016a02b739636aea3fa9e"
server_key_iv: "cd4d66de0353511d3ac2291a095a7cc3"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.392724 2190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:43.408679 2190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:43.410614 2190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:43.410760 2190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:43.411245 2190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:43.411303 2190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.411342 2190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:43.411366 2190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.418653 2190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42319
I20260621 05:15:43.418704 2320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42319 every 8 connection(s)
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:43.419057 2190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:15:43.421804 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.411693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33355 (local address 127.0.98.62:46235)
0621 05:15:43.411732 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.411745 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.411997 (+ 252us) server_negotiation.cc:407] Connection header received
0621 05:15:43.412537 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.412538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.412551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.412614 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.413404 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.414207 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.416179 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.416499 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.416581 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.419292 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.419294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.419295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.419303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.420766 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.421132 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.421133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.421133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.421179 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.421343 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.421343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.421344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.421442 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.421490 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.421693 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.421747 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:15:43.421969 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.411707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46235 (local address 127.0.98.1:33355)
0621 05:15:43.411880 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.411910 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.412408 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.412684 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.412688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.413006 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:43.413282 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.413287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.414316 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.414317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.416093 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.416096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.416624 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.416625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.416796 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.417239 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:43.417249 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:43.419193 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:43.420835 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.420840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.420841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.421068 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.421219 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.421220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.421220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.421295 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.421497 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:43.421501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:43.421599 (+ 98us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.421681 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.421787 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":115,"threads_started":1}
I20260621 05:15:43.422420 2321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46235
I20260621 05:15:43.422497 2321 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:43.422662 2321 heartbeater.cc:511] Master 127.0.98.62:46235 requested a full tablet report, sending...
I20260621 05:15:43.423136 2118 ts_manager.cc:194] Registered new tserver with Master: 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319)
I20260621 05:15:43.423975 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33355
I20260621 05:15:43.424786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 2190
I20260621 05:15:43.424862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:43.425042 392 external_mini_cluster.cc:1428] Setting key 68601dab4b0a3c8a0159bc1c8489d0b4
I20260621 05:15:43.429359 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.423399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42575 (local address 127.0.98.62:46235)
0621 05:15:43.423438 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.423439 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.423564 (+ 125us) server_negotiation.cc:407] Connection header received
0621 05:15:43.423604 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.423605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.423628 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.423698 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.424020 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.424580 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.425486 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.425759 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.425813 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.426768 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.426770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.426770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.426776 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.428245 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.428615 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.428616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.428616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.428669 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.428866 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.428866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.428867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.428950 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.429027 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.429196 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.429266 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:15:43.429373 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.423342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46235 (local address 127.0.98.1:42575)
0621 05:15:43.423489 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.423506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.423561 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.423705 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.423705 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.423905 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:43.423950 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.423951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.424677 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.424678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.425419 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.425421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.425835 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.425836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.425916 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.426282 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:43.426286 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:43.426669 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:43.428329 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.428330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.428330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.428544 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.428734 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.428735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.428735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.428804 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.429033 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:43.429034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:43.429075 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.429171 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.429267 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:15:43.449597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46235
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:43.525686 2330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:43.525874 2330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:43.525897 2330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:43.526966 2330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:43.527002 2330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:43.527022 2330 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:15:43.527034 2330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:15:43.527050 2330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:43.528465 2330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:43.528640 2330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:43.528792 2330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:43.533390 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:43.533390 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:43.533453 2330 server_base.cc:1034] running on GCE node
W20260621 05:15:43.533398 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:43.533762 2330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:43.533989 2330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:43.535148 2330 hybrid_clock.cc:648] HybridClock initialized: now 1782018943535122 us; error 37 us; skew 500 ppm
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:43.537489 2330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:15:43.537940 2330 webserver.cc:460] Webserver started at http://127.0.98.2:39765/ using document root <none> and password file <none>
I20260621 05:15:43.538160 2330 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:43.538224 2330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:43.538388 2330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:43.539337 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "713a9b164c0e4cefbf68cf5992ae51ee"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "2da22833c5ad4ea48b1908e4bddc47ca"
server_key_iv: "14207b97ff763204d6053881ed91a209"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.539754 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "713a9b164c0e4cefbf68cf5992ae51ee"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "2da22833c5ad4ea48b1908e4bddc47ca"
server_key_iv: "14207b97ff763204d6053881ed91a209"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.541065 2330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:43.541757 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.541919 2330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:43.542011 2330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "713a9b164c0e4cefbf68cf5992ae51ee"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "2da22833c5ad4ea48b1908e4bddc47ca"
server_key_iv: "14207b97ff763204d6053881ed91a209"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.542093 2330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:43.558043 2330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:43.559842 2330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:43.559991 2330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:43.560500 2330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:43.560557 2330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.560611 2330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:43.560642 2330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:43.568241 2330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35367
I20260621 05:15:43.568305 2460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35367 every 8 connection(s)
I20260621 05:15:43.568681 2330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:15:43.570621 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.560993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55965 (local address 127.0.98.62:46235)
0621 05:15:43.561063 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.561066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.561306 (+ 240us) server_negotiation.cc:407] Connection header received
0621 05:15:43.561794 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.561796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.561810 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.561906 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.562451 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.562938 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.564474 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.564704 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.564790 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.568158 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.568160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.568160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.568168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.569645 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.569974 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.569974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.569975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.570020 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.570177 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.570178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.570178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.570265 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.570308 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.570512 (+ 204us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.570563 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:15:43.570785 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.560965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46235 (local address 127.0.98.2:55965)
0621 05:15:43.561159 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.561203 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.561700 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.561955 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.561958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.562185 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:43.562355 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.562359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.563030 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.563031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.564377 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.564379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.564799 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.564800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.564961 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.565565 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:43.565572 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:43.568051 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:43.569710 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.569714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.569715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.569910 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.570060 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.570061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.570061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.570136 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.570317 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:43.570321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:43.570420 (+ 99us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.570497 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.570602 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260621 05:15:43.571214 2461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46235
I20260621 05:15:43.571307 2461 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:43.571494 2461 heartbeater.cc:511] Master 127.0.98.62:46235 requested a full tablet report, sending...
I20260621 05:15:43.571871 2117 ts_manager.cc:194] Registered new tserver with Master: 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367)
I20260621 05:15:43.572389 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55965
I20260621 05:15:43.573498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 2330
I20260621 05:15:43.573580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:43.573761 392 external_mini_cluster.cc:1428] Setting key 07880219ef87648ea13322ce97f66de0
I20260621 05:15:43.577804 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.572282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51403 (local address 127.0.98.62:46235)
0621 05:15:43.572319 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.572321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.572439 (+ 118us) server_negotiation.cc:407] Connection header received
0621 05:15:43.572477 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.572478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.572487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.572530 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.572870 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.573431 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.574268 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.574510 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.574601 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.575439 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.575440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.575440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.575447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.576700 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.577037 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.577038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.577038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.577086 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.577292 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.577293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.577293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.577427 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.577502 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.577649 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.577720 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:15:43.577817 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.572207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46235 (local address 127.0.98.2:51403)
0621 05:15:43.572366 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.572376 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.572433 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.572549 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.572549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.572737 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:43.572799 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.572801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.573521 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.573522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.574198 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.574200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.574597 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.574598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.574655 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.575077 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:43.575082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:43.575347 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:43.576792 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.576792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.576793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.576958 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.577155 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.577156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.577156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.577223 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.577494 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:43.577495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:43.577536 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.577624 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.577722 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:15:43.596517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46235
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:43.673337 2470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:43.673520 2470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:43.673543 2470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:43.674667 2470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:43.674707 2470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:43.674752 2470 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:15:43.674778 2470 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:15:43.674814 2470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:43.676301 2470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:43.676532 2470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:43.676736 2470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:43.681442 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:43.681391 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:43.681411 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:43.681542 2470 server_base.cc:1034] running on GCE node
I20260621 05:15:43.681805 2470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:43.682004 2470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:43.683159 2470 hybrid_clock.cc:648] HybridClock initialized: now 1782018943683146 us; error 31 us; skew 500 ppm
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:43.685456 2470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:15:43.685802 2470 webserver.cc:460] Webserver started at http://127.0.98.3:42125/ using document root <none> and password file <none>
I20260621 05:15:43.685978 2470 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:43.686031 2470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:43.686120 2470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:43.686972 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "38d0a110316347f5a82bee155debb7dc"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "0b03424dd38878dc21f0f527f14d2ef0"
server_key_iv: "9a937e61e3595580ee0af469d9875555"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.687233 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38d0a110316347f5a82bee155debb7dc"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "0b03424dd38878dc21f0f527f14d2ef0"
server_key_iv: "9a937e61e3595580ee0af469d9875555"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.688426 2470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:15:43.688998 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.689134 2470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.689208 2470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "38d0a110316347f5a82bee155debb7dc"
format_stamp: "Formatted at 2026-06-21 05:15:43 on dist-test-slave-czm5"
server_key: "0b03424dd38878dc21f0f527f14d2ef0"
server_key_iv: "9a937e61e3595580ee0af469d9875555"
server_key_version: "encryptionkey@0"
I20260621 05:15:43.689262 2470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:43.713941 2470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:43.716221 2470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:43.716328 2470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:43.716745 2470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:43.716782 2470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.716833 2470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:43.716858 2470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.723265 2470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34983
I20260621 05:15:43.723328 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34983 every 8 connection(s)
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:43.723764 2470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:15:43.726631 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.717208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46235 (local address 127.0.98.3:39081)
0621 05:15:43.717391 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.717428 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.717920 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.718156 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.718160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.718528 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:43.718799 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.718806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.719893 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.719894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.721273 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.721276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.721728 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.721729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.721895 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.722396 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:43.722405 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:43.724210 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:43.725530 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.725535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.725536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.725740 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.725900 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.725901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.725901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.725979 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.726158 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:43.726161 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:43.726258 (+ 97us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.726333 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.726442 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260621 05:15:43.726702 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.717235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39081 (local address 127.0.98.62:46235)
0621 05:15:43.717312 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.717315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.717502 (+ 187us) server_negotiation.cc:407] Connection header received
0621 05:15:43.718010 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.718012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.718024 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.718083 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.718916 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.719788 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.721408 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.721649 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.721724 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.724285 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.724286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.724286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.724292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.725463 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.725809 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.725810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.725810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.725859 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.726015 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.726015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.726015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.726111 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.726153 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.726515 (+ 362us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.726604 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:15:43.727067 2601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46235
I20260621 05:15:43.727156 2601 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:43.727331 2601 heartbeater.cc:511] Master 127.0.98.62:46235 requested a full tablet report, sending...
I20260621 05:15:43.727774 2117 ts_manager.cc:194] Registered new tserver with Master: 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983)
I20260621 05:15:43.728257 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39081
I20260621 05:15:43.730983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 2470
I20260621 05:15:43.731055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:43.731253 392 external_mini_cluster.cc:1428] Setting key 21296867f9a252f60bdadf0ddb6704da
I20260621 05:15:43.731937 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:15:43.733114 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.728009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46235 (local address 127.0.98.3:41573)
0621 05:15:43.728160 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.728171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.728238 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.728387 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.728388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.728574 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:43.728631 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.728633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.729217 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.729218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.729679 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.729680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.729981 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.729982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.730034 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.730350 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:43.730353 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:43.730589 (+ 236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:43.732206 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.732206 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.732207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.732356 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.732579 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.732579 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.732580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.732626 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.732882 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:43.732883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:43.732913 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.732979 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.733048 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260621 05:15:43.733232 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.728075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41573 (local address 127.0.98.62:46235)
0621 05:15:43.728159 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.728161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.728211 (+ 50us) server_negotiation.cc:407] Connection header received
0621 05:15:43.728302 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.728303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.728312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.728359 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.728706 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.729133 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.729743 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.729905 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.729982 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.730659 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.730660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.730660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.730665 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.732110 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.732433 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.732434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.732435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.732515 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.732690 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.732691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.732691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.732822 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.732887 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.733017 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.733095 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:43.753135 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.746904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.0.98.62:46235)
0621 05:15:43.746956 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.746958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.746966 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:15:43.747033 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.747033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.747042 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.747106 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.747646 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.748082 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.748836 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.749012 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.749094 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.751048 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.751049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.751049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.751054 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.752264 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.752527 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.752527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.752527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.752587 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.752781 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.752781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.752781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.752861 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.752945 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.753011 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.753060 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:15:43.754554 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:15:43.756170 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:43.761984 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.758338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35367 (local address 127.0.0.1:59942)
0621 05:15:43.758866 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.758875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.758927 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.759148 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.759151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.759153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:43.759211 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.759213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.760133 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.760133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.760923 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.760925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.761676 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.761676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.761779 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.761793 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.761861 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.761914 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":94,"threads_started":1}
I20260621 05:15:43.762017 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.758515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.0.98.2:35367)
0621 05:15:43.758820 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.758825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.758943 (+ 118us) server_negotiation.cc:407] Connection header received
0621 05:15:43.758986 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.758988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.759004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.759069 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:43.759295 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.760056 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.761043 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.761559 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.761595 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.761630 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.761893 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.761929 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":82,"threads_started":1}
I20260621 05:15:43.762777 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.758338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42319 (local address 127.0.0.1:52454)
0621 05:15:43.758688 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.758727 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.758813 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.759151 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.759152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.759153 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:43.759226 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.759228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.760499 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.760500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.761662 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.761663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.762531 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.762532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.762587 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.762591 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.762671 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.762708 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":33,"thread_start_us":63,"threads_started":1}
I20260621 05:15:43.762804 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.758443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.0.98.1:42319)
0621 05:15:43.758679 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.758684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.758800 (+ 116us) server_negotiation.cc:407] Connection header received
0621 05:15:43.758907 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.758923 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.758940 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.759031 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:43.759333 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.760405 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.761746 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.762440 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.762529 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.762580 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.762687 (+ 107us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.762719 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260621 05:15:43.762931 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.758633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.0.98.3:34983)
0621 05:15:43.758807 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.758811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.759029 (+ 218us) server_negotiation.cc:407] Connection header received
0621 05:15:43.759066 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.759068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.759083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.759149 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:43.759312 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.760410 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.761754 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.762478 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.762533 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.762583 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.762817 (+ 234us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.762849 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260621 05:15:43.762923 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 946a15e24bf0435ebd0f49f5cb14715c (DEFAULT_TABLE table=test-table [id=b61c505cc9464e2982379e83e3bbd40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:43.763067 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.758560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34983 (local address 127.0.0.1:44118)
0621 05:15:43.758927 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.758935 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.758975 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.759204 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.759205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.759206 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:43.759240 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.759241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.760499 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.760499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.761660 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.761663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.762612 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.762613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.762666 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.762670 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.762711 (+ 41us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.762749 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":54,"thread_start_us":62,"threads_started":1}
I20260621 05:15:43.763265 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946a15e24bf0435ebd0f49f5cb14715c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:43.763707 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 946a15e24bf0435ebd0f49f5cb14715c (DEFAULT_TABLE table=test-table [id=b61c505cc9464e2982379e83e3bbd40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:43.764035 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946a15e24bf0435ebd0f49f5cb14715c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:43.766909 2624 tablet_bootstrap.cc:492] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Bootstrap starting.
I20260621 05:15:43.767225 2625 tablet_bootstrap.cc:492] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: Bootstrap starting.
I20260621 05:15:43.767551 2624 tablet_bootstrap.cc:654] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:43.767656 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 946a15e24bf0435ebd0f49f5cb14715c (DEFAULT_TABLE table=test-table [id=b61c505cc9464e2982379e83e3bbd40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:43.767905 2624 log.cc:824] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:43.767985 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946a15e24bf0435ebd0f49f5cb14715c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:43.768136 2625 tablet_bootstrap.cc:654] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:43.768435 2625 log.cc:824] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:43.768671 2624 tablet_bootstrap.cc:492] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: No bootstrap required, opened a new log
I20260621 05:15:43.768756 2624 ts_tablet_manager.cc:1397] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:15:43.769238 2625 tablet_bootstrap.cc:492] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: No bootstrap required, opened a new log
I20260621 05:15:43.769311 2625 ts_tablet_manager.cc:1397] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:15:43.770319 2624 raft_consensus.cc:348] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.770484 2624 raft_consensus.cc:374] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:43.770525 2624 raft_consensus.cc:729] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713a9b164c0e4cefbf68cf5992ae51ee, State: Initialized, Role: FOLLOWER
I20260621 05:15:43.770640 2624 consensus_queue.cc:260] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.771335 2625 raft_consensus.cc:348] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.771493 2625 raft_consensus.cc:374] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:43.771536 2625 raft_consensus.cc:729] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d0a110316347f5a82bee155debb7dc, State: Initialized, Role: FOLLOWER
I20260621 05:15:43.771673 2628 tablet_bootstrap.cc:492] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Bootstrap starting.
I20260621 05:15:43.771677 2625 consensus_queue.cc:260] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.771926 2625 ts_tablet_manager.cc:1428] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:15:43.772037 2601 heartbeater.cc:503] Master 127.0.98.62:46235 was elected leader, sending a full tablet report...
I20260621 05:15:43.772063 2624 ts_tablet_manager.cc:1428] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 05:15:43.772173 2461 heartbeater.cc:503] Master 127.0.98.62:46235 was elected leader, sending a full tablet report...
I20260621 05:15:43.772323 2628 tablet_bootstrap.cc:654] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:43.772598 2628 log.cc:824] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:43.773243 2628 tablet_bootstrap.cc:492] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: No bootstrap required, opened a new log
I20260621 05:15:43.773294 2628 ts_tablet_manager.cc:1397] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:43.774679 2628 raft_consensus.cc:348] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.774797 2628 raft_consensus.cc:374] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:43.774822 2628 raft_consensus.cc:729] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b43b3dd1dcc4fa58190b6c8262a8c1b, State: Initialized, Role: FOLLOWER
I20260621 05:15:43.774961 2628 consensus_queue.cc:260] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.775207 2628 ts_tablet_manager.cc:1428] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:15:43.775269 2321 heartbeater.cc:503] Master 127.0.98.62:46235 was elected leader, sending a full tablet report...
I20260621 05:15:43.797065 2629 raft_consensus.cc:482] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:43.797145 2629 raft_consensus.cc:504] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.797444 2629 leader_election.cc:288] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367)
I20260621 05:15:43.800989 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.797668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43513 (local address 127.0.98.2:35367)
0621 05:15:43.797729 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.797731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.797739 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:15:43.797760 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.797761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.797769 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.797827 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:43.798009 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.798686 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.800137 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.800649 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.800722 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.800730 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.800895 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.800930 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:15:43.800989 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.797678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59773 (local address 127.0.98.1:42319)
0621 05:15:43.797719 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.797720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.797855 (+ 135us) server_negotiation.cc:407] Connection header received
0621 05:15:43.797891 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.797891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.797901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.797949 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:43.798093 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.798698 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.800137 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.800653 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.800722 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.800730 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.800895 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.800928 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:15:43.800997 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.797608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35367 (local address 127.0.98.3:43513)
0621 05:15:43.797659 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.797671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.797707 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.797854 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.797855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.797857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:43.797924 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.797925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.798770 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.798771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.800051 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.800054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.800732 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.800733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.800812 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.800813 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.800870 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.800911 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":90}
I20260621 05:15:43.801003 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.797608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42319 (local address 127.0.98.3:59773)
0621 05:15:43.797770 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.797788 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.797848 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.797970 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.797971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.797973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:43.798039 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.798041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.798770 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.798771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.800051 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.800054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.800733 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.800733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.800805 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.800809 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.800870 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.800911 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":71,"thread_start_us":78,"threads_started":1}
I20260621 05:15:43.801319 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946a15e24bf0435ebd0f49f5cb14715c" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" is_pre_election: true
I20260621 05:15:43.801311 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946a15e24bf0435ebd0f49f5cb14715c" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" is_pre_election: true
I20260621 05:15:43.801440 2275 raft_consensus.cc:2454] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 0.
I20260621 05:15:43.801440 2415 raft_consensus.cc:2454] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 0.
I20260621 05:15:43.801656 2488 leader_election.cc:302] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d0a110316347f5a82bee155debb7dc, 9b43b3dd1dcc4fa58190b6c8262a8c1b; no voters:
I20260621 05:15:43.801772 2629 raft_consensus.cc:2781] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:43.801833 2629 raft_consensus.cc:482] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:43.801849 2629 raft_consensus.cc:3037] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:43.802412 2629 raft_consensus.cc:504] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.802536 2629 leader_election.cc:288] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 election: Requested vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367)
I20260621 05:15:43.802644 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946a15e24bf0435ebd0f49f5cb14715c" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b"
I20260621 05:15:43.802711 2275 raft_consensus.cc:3037] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:43.802712 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946a15e24bf0435ebd0f49f5cb14715c" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713a9b164c0e4cefbf68cf5992ae51ee"
I20260621 05:15:43.802768 2415 raft_consensus.cc:3037] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:43.803536 2275 raft_consensus.cc:2454] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 1.
I20260621 05:15:43.803536 2415 raft_consensus.cc:2454] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 1.
I20260621 05:15:43.803725 2489 leader_election.cc:302] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d0a110316347f5a82bee155debb7dc, 713a9b164c0e4cefbf68cf5992ae51ee; no voters:
I20260621 05:15:43.803817 2629 raft_consensus.cc:2781] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:43.803957 2629 raft_consensus.cc:686] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 1 LEADER]: Becoming Leader. State: Replica: 38d0a110316347f5a82bee155debb7dc, State: Running, Role: LEADER
I20260621 05:15:43.804018 2629 consensus_queue.cc:237] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } }
I20260621 05:15:43.804785 2117 catalog_manager.cc:5495] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 38d0a110316347f5a82bee155debb7dc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "38d0a110316347f5a82bee155debb7dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } health_report { overall_health: UNKNOWN } } }
W20260621 05:15:43.819290 2462 tablet.cc:2369] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 05:15:43 dist-test-slave-czm5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:43.854622 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.848493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.0.98.62:46235)
0621 05:15:43.848544 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.848546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.848834 (+ 288us) server_negotiation.cc:407] Connection header received
0621 05:15:43.848899 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.848900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.848908 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.848955 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:43.849360 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.849804 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.850391 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.850564 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.850609 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.852463 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:43.852464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:43.852464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:43.852469 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:43.853716 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.854015 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.854016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.854016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.854059 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:43.854245 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:43.854246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:43.854246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:43.854344 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:43.854392 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.854501 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.854559 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:15:43.854738 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.848453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46235 (local address 127.0.0.1:58586)
0621 05:15:43.848775 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.848786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.848849 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.848983 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.848984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.849258 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:43.849302 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.849303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.849861 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.849862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.850330 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.850331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.850621 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.850621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.850658 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.851118 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:43.851122 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:43.852381 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:43.853778 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.853783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.853785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.853957 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.854109 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:43.854110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:43.854110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:43.854171 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:43.854395 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:43.854400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:43.854438 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.854488 (+ 50us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.854622 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":54,"thread_start_us":76,"threads_started":1}
I20260621 05:15:43.856504 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58586:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:15:43.856716 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:43.858147 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 42e09713987f4d00afbb7c2cc3c46411 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0d3a69157884a0499b9e370ce2feae6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:15:43.858140 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 42e09713987f4d00afbb7c2cc3c46411 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0d3a69157884a0499b9e370ce2feae6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:15:43.858315 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e09713987f4d00afbb7c2cc3c46411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:43.858314 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e09713987f4d00afbb7c2cc3c46411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:43.858362 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 42e09713987f4d00afbb7c2cc3c46411 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0d3a69157884a0499b9e370ce2feae6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:15:43.858525 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e09713987f4d00afbb7c2cc3c46411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:43.859663 2625 tablet_bootstrap.cc:492] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: Bootstrap starting.
I20260621 05:15:43.859769 2628 tablet_bootstrap.cc:492] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Bootstrap starting.
I20260621 05:15:43.859769 2624 tablet_bootstrap.cc:492] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: Bootstrap starting.
I20260621 05:15:43.860065 2625 tablet_bootstrap.cc:654] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:43.860455 2628 tablet_bootstrap.cc:654] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:43.860455 2624 tablet_bootstrap.cc:654] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:43.861119 2628 tablet_bootstrap.cc:492] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: No bootstrap required, opened a new log
I20260621 05:15:43.861176 2628 ts_tablet_manager.cc:1397] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:43.861272 2624 tablet_bootstrap.cc:492] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: No bootstrap required, opened a new log
I20260621 05:15:43.861315 2624 ts_tablet_manager.cc:1397] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:15:43.861349 2628 raft_consensus.cc:348] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.861418 2624 raft_consensus.cc:348] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.861457 2624 raft_consensus.cc:374] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:43.861449 2628 raft_consensus.cc:374] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:43.861477 2624 raft_consensus.cc:729] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713a9b164c0e4cefbf68cf5992ae51ee, State: Initialized, Role: FOLLOWER
I20260621 05:15:43.861478 2628 raft_consensus.cc:729] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b43b3dd1dcc4fa58190b6c8262a8c1b, State: Initialized, Role: FOLLOWER
I20260621 05:15:43.861524 2628 consensus_queue.cc:260] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.861505 2624 consensus_queue.cc:260] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.861632 2628 ts_tablet_manager.cc:1428] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.861641 2624 ts_tablet_manager.cc:1428] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.861680 2630 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:43.861742 2630 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:43.861881 2632 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:43.861940 2632 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:43.861964 2625 tablet_bootstrap.cc:492] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: No bootstrap required, opened a new log
I20260621 05:15:43.862015 2625 ts_tablet_manager.cc:1397] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:43.862144 2625 raft_consensus.cc:348] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.862217 2625 raft_consensus.cc:374] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:43.862257 2625 raft_consensus.cc:729] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d0a110316347f5a82bee155debb7dc, State: Initialized, Role: FOLLOWER
I20260621 05:15:43.862298 2625 consensus_queue.cc:260] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.862403 2625 ts_tablet_manager.cc:1428] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:43.862430 2629 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:43.862517 2629 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: This TxnStatusTablet replica's current role is: FOLLOWER
W20260621 05:15:43.919834 2322 tablet.cc:2369] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:15:43.974337 2602 tablet.cc:2369] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:15:43.980881 2630 raft_consensus.cc:482] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:43.980947 2630 raft_consensus.cc:504] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.981309 2630 leader_election.cc:288] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983)
I20260621 05:15:43.984203 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.981464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42319 (local address 127.0.98.2:49603)
0621 05:15:43.981538 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.981552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.981604 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.981742 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.981743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.981744 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:43.981835 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.981838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.982541 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.982542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.983316 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.983319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.983952 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.983953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.984032 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.984035 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.984085 (+ 50us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.984120 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":20}
I20260621 05:15:43.984222 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.981549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49603 (local address 127.0.98.1:42319)
0621 05:15:43.981595 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.981597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.981607 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:15:43.981661 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.981662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.981672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.981728 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:43.981923 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.982467 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.983506 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.983883 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.983935 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.983943 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.984105 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.984154 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:15:43.984315 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.981578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34983 (local address 127.0.98.2:34065)
0621 05:15:43.981727 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:43.981736 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:43.981782 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:43.981920 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:43.981921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:43.981922 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:43.981983 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.981985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.982612 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.982613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.983325 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:43.983327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.984038 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.984039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:43.984119 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.984123 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:43.984176 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:15:43.984212 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260621 05:15:43.984320 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:43.981639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34065 (local address 127.0.98.3:34983)
0621 05:15:43.981681 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:43.981683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:43.981800 (+ 117us) server_negotiation.cc:407] Connection header received
0621 05:15:43.981829 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:43.981830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:43.981841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:43.981899 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:43.982050 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.982518 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.983542 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:43.983940 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:43.984006 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:43.984016 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:43.984199 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 05:15:43.984242 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:15:43.984396 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e09713987f4d00afbb7c2cc3c46411" candidate_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" is_pre_election: true
I20260621 05:15:43.984464 2275 raft_consensus.cc:2454] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713a9b164c0e4cefbf68cf5992ae51ee in term 0.
I20260621 05:15:43.984664 2348 leader_election.cc:302] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713a9b164c0e4cefbf68cf5992ae51ee, 9b43b3dd1dcc4fa58190b6c8262a8c1b; no voters:
I20260621 05:15:43.984789 2630 raft_consensus.cc:2781] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:43.984838 2630 raft_consensus.cc:482] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:43.984864 2630 raft_consensus.cc:3037] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:43.985023 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e09713987f4d00afbb7c2cc3c46411" candidate_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d0a110316347f5a82bee155debb7dc" is_pre_election: true
I20260621 05:15:43.985137 2555 raft_consensus.cc:2454] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713a9b164c0e4cefbf68cf5992ae51ee in term 0.
I20260621 05:15:43.985692 2630 raft_consensus.cc:504] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.985810 2630 leader_election.cc:288] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [CANDIDATE]: Term 1 election: Requested vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983)
I20260621 05:15:43.985963 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e09713987f4d00afbb7c2cc3c46411" candidate_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b"
I20260621 05:15:43.985972 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e09713987f4d00afbb7c2cc3c46411" candidate_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d0a110316347f5a82bee155debb7dc"
I20260621 05:15:43.986064 2555 raft_consensus.cc:3037] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:43.986068 2275 raft_consensus.cc:3037] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:43.986627 2275 raft_consensus.cc:2454] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713a9b164c0e4cefbf68cf5992ae51ee in term 1.
I20260621 05:15:43.986690 2555 raft_consensus.cc:2454] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713a9b164c0e4cefbf68cf5992ae51ee in term 1.
I20260621 05:15:43.986840 2348 leader_election.cc:302] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d0a110316347f5a82bee155debb7dc, 713a9b164c0e4cefbf68cf5992ae51ee; no voters:
I20260621 05:15:43.986935 2630 raft_consensus.cc:2781] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:43.987049 2630 raft_consensus.cc:686] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 LEADER]: Becoming Leader. State: Replica: 713a9b164c0e4cefbf68cf5992ae51ee, State: Running, Role: LEADER
I20260621 05:15:43.987169 2630 consensus_queue.cc:237] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:43.987594 2646 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: TxnStatusTablet state changed. Reason: New leader 713a9b164c0e4cefbf68cf5992ae51ee. Latest consensus state: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:43.987702 2646 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:15:43.987877 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:15:43.987985 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:15:43.987951 2118 catalog_manager.cc:5495] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2). New cstate: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } health_report { overall_health: UNKNOWN } } }
I20260621 05:15:44.007524 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:44.005220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.0.98.2:35367)
0621 05:15:44.005278 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:44.005288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:44.005297 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:15:44.005330 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:44.005331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:44.005348 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:44.005425 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:44.005547 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.006002 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.006711 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.006895 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.006975 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:44.007152 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:44.007238 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:44.007377 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:44.007428 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 05:15:44.007446 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:15:44.007539 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:44.005122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35367 (local address 127.0.0.1:59958)
0621 05:15:44.005192 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:44.005202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:44.005267 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:44.005428 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:44.005429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:44.005430 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:15:44.005473 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:44.005476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.006096 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.006097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:44.006631 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:44.006632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.006977 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.006977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:44.007044 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:44.007055 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:15:44.007361 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:15:44.007370 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 05:15:44.007423 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:15:44.007467 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 05:15:44.009071 2555 raft_consensus.cc:1261] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Refusing update from remote peer 713a9b164c0e4cefbf68cf5992ae51ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:44.009119 2275 raft_consensus.cc:1261] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Refusing update from remote peer 713a9b164c0e4cefbf68cf5992ae51ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:44.009394 2646 consensus_queue.cc:1035] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:44.009493 2630 consensus_queue.cc:1035] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:44.010509 2629 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: TxnStatusTablet state changed. Reason: New leader 713a9b164c0e4cefbf68cf5992ae51ee. Latest consensus state: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:44.010591 2629 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:44.011452 2632 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: TxnStatusTablet state changed. Reason: New leader 713a9b164c0e4cefbf68cf5992ae51ee. Latest consensus state: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:44.011548 2632 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:44.012181 2630 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:44.012260 2630 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:15:44.012434 2629 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:44.012501 2629 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 38d0a110316347f5a82bee155debb7dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:44.012850 2646 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:44.012931 2646 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 713a9b164c0e4cefbf68cf5992ae51ee: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:15:44.013641 2654 mvcc.cc:205] Tried to move back new op lower bound from 7299149594658750464 to 7299149594571915264. Current Snapshot: MvccSnapshot[applied={T|T < 7299149594658750464}]
I20260621 05:15:44.013686 2652 mvcc.cc:205] Tried to move back new op lower bound from 7299149594658750464 to 7299149594571915264. Current Snapshot: MvccSnapshot[applied={T|T < 7299149594658750464 or (T in {7299149594658750464})}]
I20260621 05:15:44.013736 2632 tablet_replica.cc:439] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } }
I20260621 05:15:44.013821 2632 tablet_replica.cc:442] T 42e09713987f4d00afbb7c2cc3c46411 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:15:44.017103 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:44.014581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.0.98.3:34983)
0621 05:15:44.014617 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:44.014619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:44.014626 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:15:44.014701 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:44.014702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:44.014716 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:44.014768 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:44.014937 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.015398 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.016308 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.016567 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.016635 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:44.016791 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:44.016882 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:44.016965 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:44.017001 (+ 36us) server_negotiation.cc:299] Negotiation successful
0621 05:15:44.017022 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:15:44.021447 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:44.019294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48489 (local address 127.0.98.2:35367)
0621 05:15:44.019370 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:44.019372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:44.019379 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:15:44.019401 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:44.019402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:44.019411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:44.019471 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:44.019563 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.020046 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.020748 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.020945 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.021017 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:44.021204 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:44.021238 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:44.021316 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:44.021376 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 05:15:44.021391 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:15:44.021473 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:44.019228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35367 (local address 127.0.98.3:48489)
0621 05:15:44.019297 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:44.019308 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:44.019356 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:44.019472 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:44.019473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:44.019474 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:15:44.019519 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:44.019520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.020125 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.020126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:44.020666 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:44.020668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.021016 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.021017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:44.021077 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:44.021108 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:15:44.021321 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:15:44.021322 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:15:44.021372 (+ 50us) client_negotiation.cc:240] Negotiation successful
0621 05:15:44.021407 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:15:44.023386 2275 raft_consensus.cc:1261] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Refusing update from remote peer 38d0a110316347f5a82bee155debb7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:44.023388 2415 raft_consensus.cc:1261] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Refusing update from remote peer 38d0a110316347f5a82bee155debb7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:44.023634 2629 consensus_queue.cc:1035] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:44.023779 2629 consensus_queue.cc:1035] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:44.025437 2651 mvcc.cc:205] Tried to move back new op lower bound from 7299149594718208000 to 7299149593821986816. Current Snapshot: MvccSnapshot[applied={T|T < 7299149594718208000}]
I20260621 05:15:44.030918 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:44.028657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34983 (local address 127.0.98.2:36095)
0621 05:15:44.028725 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:44.028734 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:44.028773 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:44.028914 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:44.028914 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:44.028915 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:15:44.028973 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:44.028975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.029538 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.029539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:44.030082 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:44.030084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.030530 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.030531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:44.030590 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:44.030605 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:15:44.030767 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:15:44.030769 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:15:44.030811 (+ 42us) client_negotiation.cc:240] Negotiation successful
0621 05:15:44.030850 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:15:44.031081 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:44.028689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36095 (local address 127.0.98.3:34983)
0621 05:15:44.028720 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:44.028722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:44.028760 (+ 38us) server_negotiation.cc:407] Connection header received
0621 05:15:44.028829 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:44.028830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:44.028840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:44.028882 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:44.029022 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.029457 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.030178 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:44.030457 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:44.030531 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:44.030680 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:44.030709 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:44.030754 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:44.031006 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 05:15:44.031023 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:15:44.040645 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:44.040741 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:44.041479 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: Sending DeleteTablet for 3 replicas of tablet 946a15e24bf0435ebd0f49f5cb14715c
I20260621 05:15:44.041913 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 946a15e24bf0435ebd0f49f5cb14715c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52454
I20260621 05:15:44.041908 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 946a15e24bf0435ebd0f49f5cb14715c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59942
I20260621 05:15:44.042140 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 946a15e24bf0435ebd0f49f5cb14715c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260621 05:15:44.042339 2671 tablet_replica.cc:330] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: stopping tablet replica
I20260621 05:15:44.042325 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:15:44.042467 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:44.042629 2671 raft_consensus.cc:2229] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:15:44.042778 2671 raft_consensus.cc:2258] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:44.043591 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 919995df5e784a0a9d44d71ec2a75612 (DEFAULT_TABLE table=test-table [id=94feedc76571452ea4c27fe2cf9ee8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:44.043664 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 919995df5e784a0a9d44d71ec2a75612 (DEFAULT_TABLE table=test-table [id=94feedc76571452ea4c27fe2cf9ee8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:44.043726 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919995df5e784a0a9d44d71ec2a75612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:44.043769 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919995df5e784a0a9d44d71ec2a75612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:44.045154 2624 tablet_bootstrap.cc:492] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: Bootstrap starting.
I20260621 05:15:44.045177 2628 tablet_bootstrap.cc:492] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Bootstrap starting.
I20260621 05:15:44.045554 2624 tablet_bootstrap.cc:654] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:44.045579 2628 tablet_bootstrap.cc:654] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:44.046099 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 919995df5e784a0a9d44d71ec2a75612 (DEFAULT_TABLE table=test-table [id=94feedc76571452ea4c27fe2cf9ee8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:44.046207 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919995df5e784a0a9d44d71ec2a75612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:44.046495 2624 tablet_bootstrap.cc:492] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: No bootstrap required, opened a new log
I20260621 05:15:44.046561 2624 ts_tablet_manager.cc:1397] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:15:44.046711 2624 raft_consensus.cc:348] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.046797 2624 raft_consensus.cc:374] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:44.046828 2624 raft_consensus.cc:729] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713a9b164c0e4cefbf68cf5992ae51ee, State: Initialized, Role: FOLLOWER
I20260621 05:15:44.046861 2624 consensus_queue.cc:260] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.046964 2624 ts_tablet_manager.cc:1428] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:44.047484 2625 tablet_bootstrap.cc:492] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: Bootstrap starting.
I20260621 05:15:44.047910 2625 tablet_bootstrap.cc:654] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:44.048022 2628 tablet_bootstrap.cc:492] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: No bootstrap required, opened a new log
I20260621 05:15:44.048061 2628 ts_tablet_manager.cc:1397] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 05:15:44.048187 2628 raft_consensus.cc:348] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.048254 2628 raft_consensus.cc:374] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:44.048280 2628 raft_consensus.cc:729] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b43b3dd1dcc4fa58190b6c8262a8c1b, State: Initialized, Role: FOLLOWER
I20260621 05:15:44.048316 2628 consensus_queue.cc:260] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.048408 2628 ts_tablet_manager.cc:1428] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:44.049135 2671 ts_tablet_manager.cc:1905] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:44.050705 2670 tablet_replica.cc:330] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: stopping tablet replica
I20260621 05:15:44.050803 2670 raft_consensus.cc:2229] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:44.050855 2670 raft_consensus.cc:2258] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:44.050940 2671 ts_tablet_manager.cc:1918] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:44.051038 2671 log.cc:1196] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/946a15e24bf0435ebd0f49f5cb14715c
I20260621 05:15:44.051235 2670 ts_tablet_manager.cc:1905] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:44.051429 2671 ts_tablet_manager.cc:1939] T 946a15e24bf0435ebd0f49f5cb14715c P 38d0a110316347f5a82bee155debb7dc: Deleting consensus metadata
I20260621 05:15:44.051867 2104 catalog_manager.cc:4841] TS 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983): tablet 946a15e24bf0435ebd0f49f5cb14715c (table test-table [id=b61c505cc9464e2982379e83e3bbd40d]) successfully deleted
I20260621 05:15:44.052660 2670 ts_tablet_manager.cc:1918] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:44.052728 2670 log.cc:1196] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/946a15e24bf0435ebd0f49f5cb14715c
I20260621 05:15:44.052958 2670 ts_tablet_manager.cc:1939] T 946a15e24bf0435ebd0f49f5cb14715c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting consensus metadata
I20260621 05:15:44.053360 2104 catalog_manager.cc:4841] TS 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319): tablet 946a15e24bf0435ebd0f49f5cb14715c (table test-table [id=b61c505cc9464e2982379e83e3bbd40d]) successfully deleted
I20260621 05:15:44.055716 2669 tablet_replica.cc:330] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: stopping tablet replica
I20260621 05:15:44.055833 2669 raft_consensus.cc:2229] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:44.055896 2669 raft_consensus.cc:2258] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:44.056008 2625 tablet_bootstrap.cc:492] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: No bootstrap required, opened a new log
I20260621 05:15:44.056061 2625 ts_tablet_manager.cc:1397] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260621 05:15:44.056187 2625 raft_consensus.cc:348] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.056252 2625 raft_consensus.cc:374] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:44.056272 2625 raft_consensus.cc:729] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d0a110316347f5a82bee155debb7dc, State: Initialized, Role: FOLLOWER
I20260621 05:15:44.056308 2625 consensus_queue.cc:260] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.056396 2625 ts_tablet_manager.cc:1428] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:44.056895 2669 ts_tablet_manager.cc:1905] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:44.058301 2669 ts_tablet_manager.cc:1918] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:44.058378 2669 log.cc:1196] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/946a15e24bf0435ebd0f49f5cb14715c
I20260621 05:15:44.058605 2669 ts_tablet_manager.cc:1939] T 946a15e24bf0435ebd0f49f5cb14715c P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting consensus metadata
I20260621 05:15:44.058954 2105 catalog_manager.cc:4841] TS 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367): tablet 946a15e24bf0435ebd0f49f5cb14715c (table test-table [id=b61c505cc9464e2982379e83e3bbd40d]) successfully deleted
I20260621 05:15:44.443758 2664 raft_consensus.cc:482] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:44.443866 2664 raft_consensus.cc:504] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.444072 2664 leader_election.cc:288] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367)
I20260621 05:15:44.444339 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919995df5e784a0a9d44d71ec2a75612" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" is_pre_election: true
I20260621 05:15:44.444320 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919995df5e784a0a9d44d71ec2a75612" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" is_pre_election: true
I20260621 05:15:44.444429 2275 raft_consensus.cc:2454] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 0.
I20260621 05:15:44.444429 2415 raft_consensus.cc:2454] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 0.
I20260621 05:15:44.444594 2488 leader_election.cc:302] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d0a110316347f5a82bee155debb7dc, 9b43b3dd1dcc4fa58190b6c8262a8c1b; no voters:
I20260621 05:15:44.444729 2664 raft_consensus.cc:2781] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:44.444782 2664 raft_consensus.cc:482] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:44.444813 2664 raft_consensus.cc:3037] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:44.445475 2664 raft_consensus.cc:504] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.445633 2664 leader_election.cc:288] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 election: Requested vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367)
I20260621 05:15:44.445765 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919995df5e784a0a9d44d71ec2a75612" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b"
I20260621 05:15:44.445765 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919995df5e784a0a9d44d71ec2a75612" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713a9b164c0e4cefbf68cf5992ae51ee"
I20260621 05:15:44.445840 2415 raft_consensus.cc:3037] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:44.445840 2275 raft_consensus.cc:3037] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:44.446566 2415 raft_consensus.cc:2454] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 1.
I20260621 05:15:44.446566 2275 raft_consensus.cc:2454] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 1.
I20260621 05:15:44.446738 2489 leader_election.cc:302] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d0a110316347f5a82bee155debb7dc, 713a9b164c0e4cefbf68cf5992ae51ee; no voters:
I20260621 05:15:44.446847 2664 raft_consensus.cc:2781] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:44.446897 2664 raft_consensus.cc:686] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 1 LEADER]: Becoming Leader. State: Replica: 38d0a110316347f5a82bee155debb7dc, State: Running, Role: LEADER
I20260621 05:15:44.446946 2664 consensus_queue.cc:237] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:44.447530 2118 catalog_manager.cc:5495] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 38d0a110316347f5a82bee155debb7dc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "38d0a110316347f5a82bee155debb7dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } health_report { overall_health: HEALTHY } } }
I20260621 05:15:44.492233 2415 raft_consensus.cc:1261] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Refusing update from remote peer 38d0a110316347f5a82bee155debb7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:44.492312 2275 raft_consensus.cc:1261] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Refusing update from remote peer 38d0a110316347f5a82bee155debb7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:44.492455 2664 consensus_queue.cc:1035] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:44.492554 2634 consensus_queue.cc:1035] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:15:44.496977 2652 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 05:15:46.016381 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:46.016530 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:46.017428 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: Sending DeleteTablet for 3 replicas of tablet 919995df5e784a0a9d44d71ec2a75612
I20260621 05:15:46.017697 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 919995df5e784a0a9d44d71ec2a75612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52454
I20260621 05:15:46.017787 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 919995df5e784a0a9d44d71ec2a75612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59942
I20260621 05:15:46.017889 2699 tablet_replica.cc:330] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: stopping tablet replica
I20260621 05:15:46.017949 2700 tablet_replica.cc:330] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: stopping tablet replica
I20260621 05:15:46.017936 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 919995df5e784a0a9d44d71ec2a75612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260621 05:15:46.017964 2699 raft_consensus.cc:2229] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:46.018002 2699 raft_consensus.cc:2258] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:46.018008 2700 raft_consensus.cc:2229] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:46.018042 2700 raft_consensus.cc:2258] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:46.018126 2701 tablet_replica.cc:330] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: stopping tablet replica
I20260621 05:15:46.018218 2701 raft_consensus.cc:2229] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:15:46.018251 2700 ts_tablet_manager.cc:1905] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:46.018317 2701 raft_consensus.cc:2258] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:46.018345 2699 ts_tablet_manager.cc:1905] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:46.018569 2701 ts_tablet_manager.cc:1905] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:46.020117 2699 ts_tablet_manager.cc:1918] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:15:46.020243 2699 log.cc:1196] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/919995df5e784a0a9d44d71ec2a75612
I20260621 05:15:46.020532 2701 ts_tablet_manager.cc:1918] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:15:46.020571 2699 ts_tablet_manager.cc:1939] T 919995df5e784a0a9d44d71ec2a75612 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting consensus metadata
I20260621 05:15:46.020622 2701 log.cc:1196] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/919995df5e784a0a9d44d71ec2a75612
I20260621 05:15:46.020874 2701 ts_tablet_manager.cc:1939] T 919995df5e784a0a9d44d71ec2a75612 P 38d0a110316347f5a82bee155debb7dc: Deleting consensus metadata
I20260621 05:15:46.020943 2104 catalog_manager.cc:4841] TS 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319): tablet 919995df5e784a0a9d44d71ec2a75612 (table test-table [id=94feedc76571452ea4c27fe2cf9ee8bd]) successfully deleted
I20260621 05:15:46.021178 2700 ts_tablet_manager.cc:1918] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:15:46.021246 2700 log.cc:1196] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/919995df5e784a0a9d44d71ec2a75612
I20260621 05:15:46.021241 2104 catalog_manager.cc:4841] TS 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983): tablet 919995df5e784a0a9d44d71ec2a75612 (table test-table [id=94feedc76571452ea4c27fe2cf9ee8bd]) successfully deleted
I20260621 05:15:46.021445 2700 ts_tablet_manager.cc:1939] T 919995df5e784a0a9d44d71ec2a75612 P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting consensus metadata
I20260621 05:15:46.021710 2105 catalog_manager.cc:4841] TS 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367): tablet 919995df5e784a0a9d44d71ec2a75612 (table test-table [id=94feedc76571452ea4c27fe2cf9ee8bd]) successfully deleted
Jun 21 05:15:46 dist-test-slave-czm5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018946, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:15:46 dist-test-slave-czm5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018946, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:46.044366 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:46.036614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.0.98.62:46235)
0621 05:15:46.036754 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:46.036757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:46.036767 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:15:46.036848 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:46.036848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:46.036862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:46.036929 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:46.037729 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.038570 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.039737 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.039927 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.039999 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:46.042059 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:46.042060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:46.042061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:46.042066 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:46.043253 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:46.043551 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:46.043552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:46.043552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:46.043601 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:46.043838 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:46.043839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:46.043840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:46.043957 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:46.044071 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:46.044173 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 05:15:46.044253 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260621 05:15:46.046000 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:15:46.046139 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:46.047344 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 1c97c536dc0a4e51bceb86b5cc79120c (DEFAULT_TABLE table=test-table [id=35f4f4be81524539ba08f1efee7d02fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:46.047372 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 1c97c536dc0a4e51bceb86b5cc79120c (DEFAULT_TABLE table=test-table [id=35f4f4be81524539ba08f1efee7d02fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:46.047377 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 1c97c536dc0a4e51bceb86b5cc79120c (DEFAULT_TABLE table=test-table [id=35f4f4be81524539ba08f1efee7d02fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:46.047483 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c97c536dc0a4e51bceb86b5cc79120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:46.047483 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c97c536dc0a4e51bceb86b5cc79120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:46.047544 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c97c536dc0a4e51bceb86b5cc79120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:46.048633 2710 tablet_bootstrap.cc:492] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: Bootstrap starting.
I20260621 05:15:46.048822 2712 tablet_bootstrap.cc:492] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: Bootstrap starting.
I20260621 05:15:46.048988 2710 tablet_bootstrap.cc:654] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:46.049216 2712 tablet_bootstrap.cc:654] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:46.049506 2711 tablet_bootstrap.cc:492] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Bootstrap starting.
I20260621 05:15:46.049610 2710 tablet_bootstrap.cc:492] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: No bootstrap required, opened a new log
I20260621 05:15:46.049664 2710 ts_tablet_manager.cc:1397] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:46.049773 2712 tablet_bootstrap.cc:492] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: No bootstrap required, opened a new log
I20260621 05:15:46.049769 2710 raft_consensus.cc:348] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.049826 2710 raft_consensus.cc:374] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:46.049840 2710 raft_consensus.cc:729] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713a9b164c0e4cefbf68cf5992ae51ee, State: Initialized, Role: FOLLOWER
I20260621 05:15:46.049850 2712 ts_tablet_manager.cc:1397] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:46.049939 2710 consensus_queue.cc:260] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.049971 2712 raft_consensus.cc:348] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.050086 2710 ts_tablet_manager.cc:1428] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:46.050190 2712 raft_consensus.cc:374] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:46.050237 2712 raft_consensus.cc:729] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d0a110316347f5a82bee155debb7dc, State: Initialized, Role: FOLLOWER
I20260621 05:15:46.050283 2712 consensus_queue.cc:260] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.050398 2712 ts_tablet_manager.cc:1428] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 05:15:46.050688 2711 tablet_bootstrap.cc:654] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:46.051344 2711 tablet_bootstrap.cc:492] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: No bootstrap required, opened a new log
I20260621 05:15:46.051383 2711 ts_tablet_manager.cc:1397] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:46.051493 2711 raft_consensus.cc:348] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.051540 2711 raft_consensus.cc:374] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:46.051553 2711 raft_consensus.cc:729] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b43b3dd1dcc4fa58190b6c8262a8c1b, State: Initialized, Role: FOLLOWER
I20260621 05:15:46.051608 2711 consensus_queue.cc:260] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.051786 2711 ts_tablet_manager.cc:1428] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:46.092499 2629 raft_consensus.cc:482] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:46.092569 2629 raft_consensus.cc:504] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.092705 2629 leader_election.cc:288] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367)
I20260621 05:15:46.092942 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c97c536dc0a4e51bceb86b5cc79120c" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" is_pre_election: true
I20260621 05:15:46.092985 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c97c536dc0a4e51bceb86b5cc79120c" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" is_pre_election: true
I20260621 05:15:46.093041 2415 raft_consensus.cc:2454] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 0.
I20260621 05:15:46.093058 2275 raft_consensus.cc:2454] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 0.
I20260621 05:15:46.093186 2489 leader_election.cc:302] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d0a110316347f5a82bee155debb7dc, 713a9b164c0e4cefbf68cf5992ae51ee; no voters:
I20260621 05:15:46.093273 2629 raft_consensus.cc:2781] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:46.093321 2629 raft_consensus.cc:482] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:46.093340 2629 raft_consensus.cc:3037] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:46.093878 2629 raft_consensus.cc:504] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.094010 2629 leader_election.cc:288] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 election: Requested vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367)
I20260621 05:15:46.094132 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c97c536dc0a4e51bceb86b5cc79120c" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b"
I20260621 05:15:46.094213 2275 raft_consensus.cc:3037] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:46.094732 2275 raft_consensus.cc:2454] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 1.
I20260621 05:15:46.094913 2488 leader_election.cc:302] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d0a110316347f5a82bee155debb7dc, 9b43b3dd1dcc4fa58190b6c8262a8c1b; no voters:
I20260621 05:15:46.094923 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c97c536dc0a4e51bceb86b5cc79120c" candidate_uuid: "38d0a110316347f5a82bee155debb7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713a9b164c0e4cefbf68cf5992ae51ee"
I20260621 05:15:46.094975 2415 raft_consensus.cc:3037] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:46.095072 2629 raft_consensus.cc:2781] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:46.095135 2629 raft_consensus.cc:686] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 1 LEADER]: Becoming Leader. State: Replica: 38d0a110316347f5a82bee155debb7dc, State: Running, Role: LEADER
I20260621 05:15:46.095194 2629 consensus_queue.cc:237] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.095719 2118 catalog_manager.cc:5495] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 38d0a110316347f5a82bee155debb7dc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "38d0a110316347f5a82bee155debb7dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } health_report { overall_health: HEALTHY } } }
I20260621 05:15:46.095813 2415 raft_consensus.cc:2454] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d0a110316347f5a82bee155debb7dc in term 1.
I20260621 05:15:46.110559 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:46.108449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.98.3:34983)
0621 05:15:46.108593 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:46.108595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:46.108601 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:15:46.108619 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:46.108619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:46.108629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:46.108682 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:46.108794 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.109257 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.109967 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.110146 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.110195 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:46.110313 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:46.110336 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:46.110380 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:46.110439 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 05:15:46.110454 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260621 05:15:46.112602 2275 raft_consensus.cc:1261] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Refusing update from remote peer 38d0a110316347f5a82bee155debb7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:46.112632 2415 raft_consensus.cc:1261] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Refusing update from remote peer 38d0a110316347f5a82bee155debb7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:46.112852 2690 consensus_queue.cc:1035] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:46.112958 2629 consensus_queue.cc:1035] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:46.122434 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:46.122534 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:46.123070 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: Sending DeleteTablet for 3 replicas of tablet 1c97c536dc0a4e51bceb86b5cc79120c
I20260621 05:15:46.123274 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c97c536dc0a4e51bceb86b5cc79120c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59942
I20260621 05:15:46.123318 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c97c536dc0a4e51bceb86b5cc79120c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260621 05:15:46.123330 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c97c536dc0a4e51bceb86b5cc79120c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52454
I20260621 05:15:46.123354 2700 tablet_replica.cc:330] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: stopping tablet replica
I20260621 05:15:46.123409 2700 raft_consensus.cc:2229] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:46.123412 2701 tablet_replica.cc:330] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: stopping tablet replica
I20260621 05:15:46.123459 2700 raft_consensus.cc:2258] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:46.123468 2701 raft_consensus.cc:2229] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:15:46.123515 2701 raft_consensus.cc:2258] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:46.123584 2699 tablet_replica.cc:330] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: stopping tablet replica
I20260621 05:15:46.123670 2699 raft_consensus.cc:2229] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:46.123706 2699 raft_consensus.cc:2258] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:46.123687 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:15:46.123823 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:46.123910 2701 ts_tablet_manager.cc:1905] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:46.123997 2699 ts_tablet_manager.cc:1905] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:46.124071 2700 ts_tablet_manager.cc:1905] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:46.125228 2699 ts_tablet_manager.cc:1918] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:46.125264 2699 log.cc:1196] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/1c97c536dc0a4e51bceb86b5cc79120c
I20260621 05:15:46.125416 2699 ts_tablet_manager.cc:1939] T 1c97c536dc0a4e51bceb86b5cc79120c P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Deleting consensus metadata
I20260621 05:15:46.125468 2701 ts_tablet_manager.cc:1918] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:46.125519 2701 log.cc:1196] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/1c97c536dc0a4e51bceb86b5cc79120c
I20260621 05:15:46.125528 2700 ts_tablet_manager.cc:1918] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:15:46.125572 2700 log.cc:1196] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/1c97c536dc0a4e51bceb86b5cc79120c
I20260621 05:15:46.125650 2104 catalog_manager.cc:4841] TS 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319): tablet 1c97c536dc0a4e51bceb86b5cc79120c (table test-table [id=35f4f4be81524539ba08f1efee7d02fc]) successfully deleted
I20260621 05:15:46.125725 2701 ts_tablet_manager.cc:1939] T 1c97c536dc0a4e51bceb86b5cc79120c P 38d0a110316347f5a82bee155debb7dc: Deleting consensus metadata
I20260621 05:15:46.125766 2700 ts_tablet_manager.cc:1939] T 1c97c536dc0a4e51bceb86b5cc79120c P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting consensus metadata
I20260621 05:15:46.126081 2104 catalog_manager.cc:4841] TS 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983): tablet 1c97c536dc0a4e51bceb86b5cc79120c (table test-table [id=35f4f4be81524539ba08f1efee7d02fc]) successfully deleted
I20260621 05:15:46.126096 2105 catalog_manager.cc:4841] TS 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367): tablet 1c97c536dc0a4e51bceb86b5cc79120c (table test-table [id=35f4f4be81524539ba08f1efee7d02fc]) successfully deleted
I20260621 05:15:46.127069 2394 tablet_service.cc:1467] Processing CreateTablet for tablet a36e76c79de54460baf2dbdd907d9fd9 (DEFAULT_TABLE table=test-table [id=27a62e6c6c8c4c898d4fd52b62b18786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:46.127110 2255 tablet_service.cc:1467] Processing CreateTablet for tablet a36e76c79de54460baf2dbdd907d9fd9 (DEFAULT_TABLE table=test-table [id=27a62e6c6c8c4c898d4fd52b62b18786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:46.127197 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36e76c79de54460baf2dbdd907d9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:46.127211 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36e76c79de54460baf2dbdd907d9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:46.128145 2711 tablet_bootstrap.cc:492] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Bootstrap starting.
I20260621 05:15:46.128114 2710 tablet_bootstrap.cc:492] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: Bootstrap starting.
I20260621 05:15:46.128185 2535 tablet_service.cc:1467] Processing CreateTablet for tablet a36e76c79de54460baf2dbdd907d9fd9 (DEFAULT_TABLE table=test-table [id=27a62e6c6c8c4c898d4fd52b62b18786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:46.128268 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36e76c79de54460baf2dbdd907d9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:46.128419 2710 tablet_bootstrap.cc:654] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:46.128692 2711 tablet_bootstrap.cc:654] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:46.128928 2710 tablet_bootstrap.cc:492] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: No bootstrap required, opened a new log
I20260621 05:15:46.128961 2710 ts_tablet_manager.cc:1397] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:46.129081 2710 raft_consensus.cc:348] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.129154 2710 raft_consensus.cc:374] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:46.129177 2710 raft_consensus.cc:729] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713a9b164c0e4cefbf68cf5992ae51ee, State: Initialized, Role: FOLLOWER
I20260621 05:15:46.129215 2710 consensus_queue.cc:260] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.129326 2710 ts_tablet_manager.cc:1428] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:46.129366 2711 tablet_bootstrap.cc:492] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: No bootstrap required, opened a new log
I20260621 05:15:46.129406 2711 ts_tablet_manager.cc:1397] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:46.129561 2712 tablet_bootstrap.cc:492] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: Bootstrap starting.
I20260621 05:15:46.129545 2711 raft_consensus.cc:348] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.129609 2711 raft_consensus.cc:374] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:46.129631 2711 raft_consensus.cc:729] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b43b3dd1dcc4fa58190b6c8262a8c1b, State: Initialized, Role: FOLLOWER
I20260621 05:15:46.129669 2711 consensus_queue.cc:260] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.129748 2711 ts_tablet_manager.cc:1428] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:46.129822 2712 tablet_bootstrap.cc:654] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:46.130522 2712 tablet_bootstrap.cc:492] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: No bootstrap required, opened a new log
I20260621 05:15:46.130574 2712 ts_tablet_manager.cc:1397] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:46.130693 2712 raft_consensus.cc:348] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.130764 2712 raft_consensus.cc:374] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:46.130795 2712 raft_consensus.cc:729] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d0a110316347f5a82bee155debb7dc, State: Initialized, Role: FOLLOWER
I20260621 05:15:46.130848 2712 consensus_queue.cc:260] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.130944 2712 ts_tablet_manager.cc:1428] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:46.320915 2685 raft_consensus.cc:482] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:46.321008 2685 raft_consensus.cc:504] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.321192 2685 leader_election.cc:288] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983)
I20260621 05:15:46.321456 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e76c79de54460baf2dbdd907d9fd9" candidate_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" is_pre_election: true
I20260621 05:15:46.321480 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e76c79de54460baf2dbdd907d9fd9" candidate_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d0a110316347f5a82bee155debb7dc" is_pre_election: true
I20260621 05:15:46.321565 2275 raft_consensus.cc:2454] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713a9b164c0e4cefbf68cf5992ae51ee in term 0.
I20260621 05:15:46.321593 2555 raft_consensus.cc:2454] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713a9b164c0e4cefbf68cf5992ae51ee in term 0.
I20260621 05:15:46.321753 2348 leader_election.cc:302] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713a9b164c0e4cefbf68cf5992ae51ee, 9b43b3dd1dcc4fa58190b6c8262a8c1b; no voters:
I20260621 05:15:46.321861 2685 raft_consensus.cc:2781] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:46.321929 2685 raft_consensus.cc:482] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:46.321960 2685 raft_consensus.cc:3037] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:46.322750 2685 raft_consensus.cc:504] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.322887 2685 leader_election.cc:288] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [CANDIDATE]: Term 1 election: Requested vote from peers 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319), 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983)
I20260621 05:15:46.323045 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e76c79de54460baf2dbdd907d9fd9" candidate_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d0a110316347f5a82bee155debb7dc"
I20260621 05:15:46.323045 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e76c79de54460baf2dbdd907d9fd9" candidate_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b"
I20260621 05:15:46.323124 2275 raft_consensus.cc:3037] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:46.323125 2555 raft_consensus.cc:3037] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:46.323830 2275 raft_consensus.cc:2454] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713a9b164c0e4cefbf68cf5992ae51ee in term 1.
I20260621 05:15:46.323890 2555 raft_consensus.cc:2454] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713a9b164c0e4cefbf68cf5992ae51ee in term 1.
I20260621 05:15:46.324003 2348 leader_election.cc:302] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713a9b164c0e4cefbf68cf5992ae51ee, 9b43b3dd1dcc4fa58190b6c8262a8c1b; no voters:
I20260621 05:15:46.324123 2685 raft_consensus.cc:2781] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:46.324206 2685 raft_consensus.cc:686] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 LEADER]: Becoming Leader. State: Replica: 713a9b164c0e4cefbf68cf5992ae51ee, State: Running, Role: LEADER
I20260621 05:15:46.324257 2685 consensus_queue.cc:237] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260621 05:15:46.324859 2117 catalog_manager.cc:5495] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2). New cstate: current_term: 1 leader_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713a9b164c0e4cefbf68cf5992ae51ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } health_report { overall_health: UNKNOWN } } }
I20260621 05:15:46.358753 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:46.356662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.0.98.2:35367)
0621 05:15:46.356825 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:46.356827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:46.356833 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:15:46.356850 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:46.356851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:46.356860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:46.356916 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:46.357025 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.357485 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.358142 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.358326 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.358373 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:46.358526 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:46.358550 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:46.358587 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:46.358638 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 05:15:46.358652 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260621 05:15:46.362852 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:46.360590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50669 (local address 127.0.98.2:35367)
0621 05:15:46.360650 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:46.360652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:46.360739 (+ 87us) server_negotiation.cc:407] Connection header received
0621 05:15:46.360800 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:46.360801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:46.360813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:46.360867 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:46.360982 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.361419 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.362129 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.362390 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.362473 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:46.362614 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:46.362640 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:46.362685 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:46.362732 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 05:15:46.362780 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":41}
I20260621 05:15:46.362855 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:46.360506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35367 (local address 127.0.98.2:50669)
0621 05:15:46.360665 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:46.360675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:46.360752 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:46.360870 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:46.360870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:46.360872 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:15:46.360918 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:46.360919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.361498 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.361499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:46.362047 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:46.362048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:46.362473 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:46.362473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:46.362521 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:46.362524 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:15:46.362683 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:15:46.362685 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:15:46.362730 (+ 45us) client_negotiation.cc:240] Negotiation successful
0621 05:15:46.362789 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":14,"thread_start_us":98,"threads_started":1}
I20260621 05:15:46.364638 2275 raft_consensus.cc:1261] T a36e76c79de54460baf2dbdd907d9fd9 P 9b43b3dd1dcc4fa58190b6c8262a8c1b [term 1 FOLLOWER]: Refusing update from remote peer 713a9b164c0e4cefbf68cf5992ae51ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:46.364672 2555 raft_consensus.cc:1261] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Refusing update from remote peer 713a9b164c0e4cefbf68cf5992ae51ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:46.364852 2630 consensus_queue.cc:1035] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d0a110316347f5a82bee155debb7dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:46.364962 2680 consensus_queue.cc:1035] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b43b3dd1dcc4fa58190b6c8262a8c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:15:46.369338 2723 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 05:15:47.889222 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:47.889333 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:47.890034 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff41bd597344307baabf1dbf0927c46: Sending DeleteTablet for 3 replicas of tablet a36e76c79de54460baf2dbdd907d9fd9
I20260621 05:15:47.890265 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet a36e76c79de54460baf2dbdd907d9fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59942
I20260621 05:15:47.890275 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet a36e76c79de54460baf2dbdd907d9fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260621 05:15:47.890468 2758 tablet_replica.cc:330] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: stopping tablet replica
I20260621 05:15:47.890563 2758 raft_consensus.cc:2229] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:47.890573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 2190
I20260621 05:15:47.890605 2758 raft_consensus.cc:2258] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:47.890820 2758 ts_tablet_manager.cc:1905] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:47.891701 2757 tablet_replica.cc:330] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: stopping tablet replica
I20260621 05:15:47.891819 2757 raft_consensus.cc:2229] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:15:47.891965 2757 raft_consensus.cc:2258] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:47.892277 2757 ts_tablet_manager.cc:1905] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:47.892630 2758 ts_tablet_manager.cc:1918] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:15:47.892732 2758 log.cc:1196] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/a36e76c79de54460baf2dbdd907d9fd9
I20260621 05:15:47.892987 2758 ts_tablet_manager.cc:1939] T a36e76c79de54460baf2dbdd907d9fd9 P 38d0a110316347f5a82bee155debb7dc: Deleting consensus metadata
I20260621 05:15:47.893306 2104 catalog_manager.cc:4841] TS 38d0a110316347f5a82bee155debb7dc (127.0.98.3:34983): tablet a36e76c79de54460baf2dbdd907d9fd9 (table test-table [id=27a62e6c6c8c4c898d4fd52b62b18786]) successfully deleted
I20260621 05:15:47.893877 2757 ts_tablet_manager.cc:1918] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:15:47.893949 2757 log.cc:1196] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/a36e76c79de54460baf2dbdd907d9fd9
I20260621 05:15:47.894155 2757 ts_tablet_manager.cc:1939] T a36e76c79de54460baf2dbdd907d9fd9 P 713a9b164c0e4cefbf68cf5992ae51ee: Deleting consensus metadata
I20260621 05:15:47.894480 2105 catalog_manager.cc:4841] TS 713a9b164c0e4cefbf68cf5992ae51ee (127.0.98.2:35367): tablet a36e76c79de54460baf2dbdd907d9fd9 (table test-table [id=27a62e6c6c8c4c898d4fd52b62b18786]) successfully deleted
W20260621 05:15:47.895825 2104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42319 (error 108)
I20260621 05:15:47.896359 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:47.896171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42319 (local address 127.0.0.1:52468)
0621 05:15:47.896283 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:47.896330 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
I20260621 05:15:47.896447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 2330
W20260621 05:15:47.896474 2104 catalog_manager.cc:4584] TS 9b43b3dd1dcc4fa58190b6c8262a8c1b (127.0.98.1:42319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a36e76c79de54460baf2dbdd907d9fd9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42319: connect: Connection refused (error 111)
I20260621 05:15:47.902519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 2470
I20260621 05:15:47.907521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 2086
2026-06-21T05:15:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7679 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:47 dist-test-slave-czm5 krb5kdc[2763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:47 dist-test-slave-czm5 krb5kdc[2763](info): set up 2 sockets
Jun 21 05:15:47 dist-test-slave-czm5 krb5kdc[2763](info): commencing operation
krb5kdc: starting...
W20260621 05:15:48.935238 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:49.058020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43901
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:49.132138 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:49.132313 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:49.132333 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:49.133350 2776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:49.133380 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:49.133394 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:49.133405 2776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:49.133416 2776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:49.134788 2776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:49.134958 2776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:49.135106 2776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:49.139547 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:49.139536 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:49.139534 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:49.139776 2776 server_base.cc:1034] running on GCE node
I20260621 05:15:49.140056 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:49.140328 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:49.141476 2776 hybrid_clock.cc:648] HybridClock initialized: now 1782018949141456 us; error 34 us; skew 500 ppm
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:49.143713 2776 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:49.144102 2776 webserver.cc:460] Webserver started at http://127.0.98.62:33453/ using document root <none> and password file <none>
I20260621 05:15:49.144286 2776 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:49.144323 2776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:49.144414 2776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:49.145277 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "cda5d7e593b94f6cb3b759bba44af7c0"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "74b481f07a619e5e5760bff5ab967a6d"
server_key_iv: "b50ec9ce772c5b25d0c928946153fc51"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.145576 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "cda5d7e593b94f6cb3b759bba44af7c0"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "74b481f07a619e5e5760bff5ab967a6d"
server_key_iv: "b50ec9ce772c5b25d0c928946153fc51"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.146764 2776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:15:49.147338 2791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.147475 2776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.147547 2776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "cda5d7e593b94f6cb3b759bba44af7c0"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "74b481f07a619e5e5760bff5ab967a6d"
server_key_iv: "b50ec9ce772c5b25d0c928946153fc51"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.147595 2776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:49.161029 2776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:49.162607 2776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:49.162706 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:49.166111 2776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43901
I20260621 05:15:49.166155 2843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43901 every 8 connection(s)
I20260621 05:15:49.166599 2776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:49.167224 2844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:49.169248 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0: Bootstrap starting.
I20260621 05:15:49.169806 2844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:49.170073 2844 log.cc:824] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:49.170673 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0: No bootstrap required, opened a new log
I20260621 05:15:49.171795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 2776
I20260621 05:15:49.171907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:49.172097 392 external_mini_cluster.cc:1428] Setting key 5e9eabda504bb4747d4a95df81bc5047
I20260621 05:15:49.172284 2844 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5d7e593b94f6cb3b759bba44af7c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43901 } }
I20260621 05:15:49.172474 2844 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:49.172523 2844 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda5d7e593b94f6cb3b759bba44af7c0, State: Initialized, Role: FOLLOWER
I20260621 05:15:49.172639 2844 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5d7e593b94f6cb3b759bba44af7c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43901 } }
I20260621 05:15:49.172704 2844 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:49.172725 2844 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:49.172770 2844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:49.173355 2844 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5d7e593b94f6cb3b759bba44af7c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43901 } }
I20260621 05:15:49.173457 2844 leader_election.cc:302] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda5d7e593b94f6cb3b759bba44af7c0; no voters:
I20260621 05:15:49.173635 2844 leader_election.cc:288] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:49.173663 2849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:49.173756 2849 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [term 1 LEADER]: Becoming Leader. State: Replica: cda5d7e593b94f6cb3b759bba44af7c0, State: Running, Role: LEADER
I20260621 05:15:49.173866 2844 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:49.173828 2849 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5d7e593b94f6cb3b759bba44af7c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43901 } }
I20260621 05:15:49.174131 2849 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda5d7e593b94f6cb3b759bba44af7c0. Latest consensus state: current_term: 1 leader_uuid: "cda5d7e593b94f6cb3b759bba44af7c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5d7e593b94f6cb3b759bba44af7c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43901 } } }
I20260621 05:15:49.174199 2849 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:49.174141 2851 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda5d7e593b94f6cb3b759bba44af7c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5d7e593b94f6cb3b759bba44af7c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43901 } } }
I20260621 05:15:49.174382 2851 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:49.174849 2853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:49.175314 2853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:49.178347 2853 catalog_manager.cc:1347] Generated new cluster ID: 74bb8116c99243cea3145060232a3cb7
I20260621 05:15:49.178408 2853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:49.182770 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:49.172524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.0.98.62:43901)
0621 05:15:49.173096 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:49.173105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:49.173160 (+ 55us) server_negotiation.cc:407] Connection header received
0621 05:15:49.173279 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:49.173285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:49.173339 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:49.173479 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:49.174211 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.175608 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.176466 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.176938 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.177062 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:49.179452 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:49.179456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:49.179456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:49.179465 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:49.181278 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:49.181677 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:49.181679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:49.181679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:49.181791 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:49.181979 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:49.181979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:49.181979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:49.182242 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:49.182385 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:49.182499 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 05:15:49.182607 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":476,"threads_started":1}
I20260621 05:15:49.191610 2853 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:49.192271 2853 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:49.203230 2853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda5d7e593b94f6cb3b759bba44af7c0: Generated new TSK 0
I20260621 05:15:49.203382 2853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:15:49.232316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43901
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:49.307818 2870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:49.308027 2870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:49.308063 2870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:49.309110 2870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:49.309166 2870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:49.309204 2870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:49.310560 2870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:49.310786 2870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:49.310986 2870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:49.315445 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:49.315456 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:49.315464 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:49.315730 2870 server_base.cc:1034] running on GCE node
I20260621 05:15:49.316104 2870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:49.316334 2870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:49.317489 2870 hybrid_clock.cc:648] HybridClock initialized: now 1782018949317475 us; error 29 us; skew 500 ppm
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:49.319824 2870 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:49.320201 2870 webserver.cc:460] Webserver started at http://127.0.98.1:33071/ using document root <none> and password file <none>
I20260621 05:15:49.320393 2870 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:49.320441 2870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:49.320570 2870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:49.321471 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "3efd4311127f4bca957f5824c0085754"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "6a33eb5e774b29a479158aabc56b7856"
server_key_iv: "0c7ccf5e836f185420c92a331a8d2a3d"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.321786 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3efd4311127f4bca957f5824c0085754"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "6a33eb5e774b29a479158aabc56b7856"
server_key_iv: "0c7ccf5e836f185420c92a331a8d2a3d"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.322937 2870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:15:49.323544 2885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.323691 2870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:49.323776 2870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "3efd4311127f4bca957f5824c0085754"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "6a33eb5e774b29a479158aabc56b7856"
server_key_iv: "0c7ccf5e836f185420c92a331a8d2a3d"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.323841 2870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:49.361047 2870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:49.362636 2870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:49.362802 2870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:49.363010 2870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:49.363302 2870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:49.363349 2870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.363380 2870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:49.363404 2870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.368855 2870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35321
I20260621 05:15:49.368901 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35321 every 8 connection(s)
I20260621 05:15:49.369179 2870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:49.376706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 2870
I20260621 05:15:49.376817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:49.377013 392 external_mini_cluster.cc:1428] Setting key 4019c1745d61038e533fa081ef41527c
I20260621 05:15:49.377651 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:49.369792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38989 (local address 127.0.98.62:43901)
0621 05:15:49.369851 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:49.369853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:49.370051 (+ 198us) server_negotiation.cc:407] Connection header received
0621 05:15:49.370934 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:49.370935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:49.370944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:49.371003 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:49.371527 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.372005 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.372873 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.373051 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.373098 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:49.374969 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:49.374970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:49.374971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:49.374978 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:49.376267 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:49.376693 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:49.376694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:49.376694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:49.376740 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:49.376985 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:49.376986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:49.376987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:49.377125 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:49.377188 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:49.377469 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 05:15:49.377545 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:15:49.377961 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:49.369758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43901 (local address 127.0.98.1:38989)
0621 05:15:49.369925 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:49.369962 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:49.370850 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:49.371057 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:49.371060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:49.371289 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:49.371465 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:49.371469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.372068 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.372068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:49.372807 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:49.372809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.373113 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.373114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:49.373201 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:49.373580 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:49.373586 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:49.374886 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:49.376336 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:49.376341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:49.376343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:49.376609 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:49.376785 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:49.376786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:49.376786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:49.376913 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:49.377172 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:49.377178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:49.377337 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 05:15:49.377444 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:15:49.377607 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260621 05:15:49.378402 2999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43901
I20260621 05:15:49.378491 2999 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:49.378660 2999 heartbeater.cc:511] Master 127.0.98.62:43901 requested a full tablet report, sending...
I20260621 05:15:49.379099 2808 ts_manager.cc:194] Registered new tserver with Master: 3efd4311127f4bca957f5824c0085754 (127.0.98.1:35321)
I20260621 05:15:49.379820 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38989
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:15:49.398582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43901
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:49.475085 3004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:49.475260 3004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:49.475281 3004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:49.476461 3004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:49.476500 3004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:49.476547 3004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:49.478001 3004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:49.478205 3004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:49.478410 3004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:49.482910 3012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:49.483088 3004 server_base.cc:1034] running on GCE node
W20260621 05:15:49.482921 3009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:49.482919 3010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:49.483417 3004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:49.483675 3004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:49.484822 3004 hybrid_clock.cc:648] HybridClock initialized: now 1782018949484815 us; error 37 us; skew 500 ppm
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:49.487097 3004 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:15:49.487558 3004 webserver.cc:460] Webserver started at http://127.0.98.2:36031/ using document root <none> and password file <none>
I20260621 05:15:49.487773 3004 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:49.487834 3004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:49.487958 3004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:49.488852 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "d95678c3ca534b6eacbbed73dabc404e"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "36ca7929ad64f5f8d13723a195d10326"
server_key_iv: "f290f94af93fed5531604d0607dea9a5"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.489234 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d95678c3ca534b6eacbbed73dabc404e"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "36ca7929ad64f5f8d13723a195d10326"
server_key_iv: "f290f94af93fed5531604d0607dea9a5"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.490437 3004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:15:49.491053 3019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.491206 3004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:49.491291 3004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "d95678c3ca534b6eacbbed73dabc404e"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "36ca7929ad64f5f8d13723a195d10326"
server_key_iv: "f290f94af93fed5531604d0607dea9a5"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.491371 3004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:49.502234 3004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:49.503800 3004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:49.503933 3004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:49.504114 3004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:49.504396 3004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:49.504444 3004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.504483 3004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:49.504506 3004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.509958 3004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35637
I20260621 05:15:49.509992 3132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35637 every 8 connection(s)
I20260621 05:15:49.510263 3004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:15:49.512578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 3004
I20260621 05:15:49.512674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:49.512883 392 external_mini_cluster.cc:1428] Setting key 1ce05303874edfd2fb1d098bbffb290c
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:49.519986 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:49.510913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41315 (local address 127.0.98.62:43901)
0621 05:15:49.510987 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:49.510989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:49.511142 (+ 153us) server_negotiation.cc:407] Connection header received
0621 05:15:49.512063 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:49.512064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:49.512073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:49.512153 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:49.512767 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.513451 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.514463 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.514773 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.514843 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:49.517366 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:49.517368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:49.517368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:49.517374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:49.518684 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:49.519129 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:49.519130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:49.519130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:49.519174 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:49.519415 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:49.519415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:49.519416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:49.519507 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:49.519551 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:49.519870 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 05:15:49.519927 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260621 05:15:49.520228 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:49.510858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43901 (local address 127.0.98.2:41315)
0621 05:15:49.511025 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:49.511054 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:49.511963 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:49.512173 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:49.512176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:49.512460 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:49.512630 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:49.512634 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.513521 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.513523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:49.514380 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:49.514382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.514861 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.514863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:49.515022 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:49.515493 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:49.515502 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:49.517282 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:49.518770 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:49.518776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:49.518778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:49.519057 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:49.519233 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:49.519233 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:49.519234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:49.519357 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:49.519565 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:49.519570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:49.519733 (+ 163us) client_negotiation.cc:769] Sending connection context
0621 05:15:49.519858 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 05:15:49.520008 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260621 05:15:49.520680 3133 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43901
I20260621 05:15:49.520759 3133 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260621 05:15:49.520920 3133 heartbeater.cc:511] Master 127.0.98.62:43901 requested a full tablet report, sending...
I20260621 05:15:49.521256 2808 ts_manager.cc:194] Registered new tserver with Master: d95678c3ca534b6eacbbed73dabc404e (127.0.98.2:35637)
I20260621 05:15:49.521584 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:15:49.536099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43901
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:49.608292 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:49.608498 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:49.608533 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:49.609606 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:49.609663 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:49.609701 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:49.611105 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:49.611349 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:49.611564 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:49.616051 3143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:49.616037 3144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:49.616134 3138 server_base.cc:1034] running on GCE node
W20260621 05:15:49.616073 3146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:49.616459 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:49.616701 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:49.617831 3138 hybrid_clock.cc:648] HybridClock initialized: now 1782018949617813 us; error 36 us; skew 500 ppm
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:49.619976 3138 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:15:49.620357 3138 webserver.cc:460] Webserver started at http://127.0.98.3:37059/ using document root <none> and password file <none>
I20260621 05:15:49.620522 3138 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:49.620556 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:49.620631 3138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:49.621507 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "aa8e1016c22d4ab5a83154e10848502c"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "62ea5d5d31ee942be8e62c37cb6e304a"
server_key_iv: "168de39e8514db4bf737574c33d9dd39"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.621776 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aa8e1016c22d4ab5a83154e10848502c"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "62ea5d5d31ee942be8e62c37cb6e304a"
server_key_iv: "168de39e8514db4bf737574c33d9dd39"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.622948 3138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:15:49.623486 3153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.623660 3138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.623740 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "aa8e1016c22d4ab5a83154e10848502c"
format_stamp: "Formatted at 2026-06-21 05:15:49 on dist-test-slave-czm5"
server_key: "62ea5d5d31ee942be8e62c37cb6e304a"
server_key_iv: "168de39e8514db4bf737574c33d9dd39"
server_key_version: "encryptionkey@0"
I20260621 05:15:49.623804 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:49.633159 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:49.634768 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:49.634917 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:49.635113 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:49.635387 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:49.635442 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.635479 3138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:49.635509 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:49.641364 3138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35873
I20260621 05:15:49.641395 3266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35873 every 8 connection(s)
I20260621 05:15:49.641702 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:49.649947 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:49.642349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42117 (local address 127.0.98.62:43901)
0621 05:15:49.642391 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:49.642393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:49.642586 (+ 193us) server_negotiation.cc:407] Connection header received
0621 05:15:49.643473 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:49.643475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:49.643487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:49.643537 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:49.644132 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.644655 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.645613 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.645812 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.645894 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:49.647783 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:49.647785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:49.647785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:49.647791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:49.648813 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:49.649145 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:49.649145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:49.649146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:49.649188 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:49.649361 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:49.649362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:49.649362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:49.649484 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:49.649531 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:49.649773 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 05:15:49.649846 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:15:49.650012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 3138
I20260621 05:15:49.650108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:49.650103 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:49.642311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43901 (local address 127.0.98.3:42117)
0621 05:15:49.642449 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:49.642514 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:49.643368 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:49.643608 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:49.643611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:49.643851 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:49.644034 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:49.644038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.644751 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.644752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:49.645527 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:49.645529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:49.645902 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:49.645903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:49.646021 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:49.646416 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:49.646423 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:49.647699 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:49.648873 (+ 1174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:49.648878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:49.648879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:49.649090 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:49.649236 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:49.649236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:49.649236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:49.649316 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:49.649531 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:49.649534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:49.649661 (+ 127us) client_negotiation.cc:769] Sending connection context
0621 05:15:49.649751 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 05:15:49.649875 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260621 05:15:49.650296 392 external_mini_cluster.cc:1428] Setting key 48c077771bc4be01c2cc061de1441a60
I20260621 05:15:49.650578 3267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43901
I20260621 05:15:49.650671 3267 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:49.650840 3267 heartbeater.cc:511] Master 127.0.98.62:43901 requested a full tablet report, sending...
I20260621 05:15:49.651268 2808 ts_manager.cc:194] Registered new tserver with Master: aa8e1016c22d4ab5a83154e10848502c (127.0.98.3:35873)
I20260621 05:15:49.651577 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42117
I20260621 05:15:49.661788 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:15:49.671651 2848 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:15:49.670952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.98.62:43901)
0621 05:15:49.670997 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:49.670998 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:49.671090 (+ 92us) server_negotiation.cc:407] Connection header received
0621 05:15:49.671142 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:49.671143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:49.671150 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:49.671188 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:49.671566 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53370: BlockingRecv error: recv got EOF from 127.0.0.1:53370 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:15:49.672541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 2870
I20260621 05:15:49.676730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 3004
I20260621 05:15:49.680490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 3138
I20260621 05:15:49.684360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 2776
2026-06-21T05:15:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1775 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[3279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[3279](info): set up 2 sockets
Jun 21 05:15:49 dist-test-slave-czm5 krb5kdc[3279](info): commencing operation
krb5kdc: starting...
W20260621 05:15:50.683267 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:50 dist-test-slave-czm5 krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:50.790490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42229
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:50.865765 3292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:50.865944 3292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:50.865965 3292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:50.867022 3292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:50.867054 3292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:50.867069 3292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:50.867080 3292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:50.867091 3292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:50.868455 3292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:50.868616 3292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:50.868757 3292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:50.873047 3298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:50.873064 3297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:50.873080 3300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:50.873257 3292 server_base.cc:1034] running on GCE node
I20260621 05:15:50.873454 3292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:50.873659 3292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:50.874819 3292 hybrid_clock.cc:648] HybridClock initialized: now 1782018950874799 us; error 25 us; skew 500 ppm
Jun 21 05:15:50 dist-test-slave-czm5 krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:50.877147 3292 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:50.877493 3292 webserver.cc:460] Webserver started at http://127.0.98.62:40209/ using document root <none> and password file <none>
I20260621 05:15:50.877667 3292 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:50.877703 3292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:50.877800 3292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:50.878705 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "8a23767e317b4e0e8572dd3c60efc41b"
format_stamp: "Formatted at 2026-06-21 05:15:50 on dist-test-slave-czm5"
server_key: "290d10be760403c57782de5d33df5880"
server_key_iv: "5dcd4b70fde61cc4ff011b63c2cf2805"
server_key_version: "encryptionkey@0"
I20260621 05:15:50.878971 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a23767e317b4e0e8572dd3c60efc41b"
format_stamp: "Formatted at 2026-06-21 05:15:50 on dist-test-slave-czm5"
server_key: "290d10be760403c57782de5d33df5880"
server_key_iv: "5dcd4b70fde61cc4ff011b63c2cf2805"
server_key_version: "encryptionkey@0"
I20260621 05:15:50.880218 3292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:50.880849 3307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:50.881006 3292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:50.881078 3292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "8a23767e317b4e0e8572dd3c60efc41b"
format_stamp: "Formatted at 2026-06-21 05:15:50 on dist-test-slave-czm5"
server_key: "290d10be760403c57782de5d33df5880"
server_key_iv: "5dcd4b70fde61cc4ff011b63c2cf2805"
server_key_version: "encryptionkey@0"
I20260621 05:15:50.881126 3292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:50.897593 3292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:50.899231 3292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:50.899338 3292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:50.902865 3292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42229
I20260621 05:15:50.902925 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42229 every 8 connection(s)
I20260621 05:15:50.903213 3292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:50.904052 3360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:50.904863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 3292
I20260621 05:15:50.904933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:50.905088 392 external_mini_cluster.cc:1428] Setting key 03273a945c2e29ef5da8f47719f572aa
I20260621 05:15:50.906394 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b: Bootstrap starting.
I20260621 05:15:50.906981 3360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:50.907292 3360 log.cc:824] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:50.907899 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b: No bootstrap required, opened a new log
I20260621 05:15:50.909451 3360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a23767e317b4e0e8572dd3c60efc41b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } }
I20260621 05:15:50.909582 3360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:50.909622 3360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a23767e317b4e0e8572dd3c60efc41b, State: Initialized, Role: FOLLOWER
I20260621 05:15:50.909747 3360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a23767e317b4e0e8572dd3c60efc41b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } }
I20260621 05:15:50.909823 3360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:50.909862 3360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:50.909905 3360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:50.910370 3360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a23767e317b4e0e8572dd3c60efc41b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } }
I20260621 05:15:50.910480 3360 leader_election.cc:302] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a23767e317b4e0e8572dd3c60efc41b; no voters:
I20260621 05:15:50.910624 3360 leader_election.cc:288] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:50.910676 3365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:50.910808 3365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [term 1 LEADER]: Becoming Leader. State: Replica: 8a23767e317b4e0e8572dd3c60efc41b, State: Running, Role: LEADER
I20260621 05:15:50.910849 3360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [sys.catalog]: configured and running, proceeding with master startup.
Jun 21 05:15:50 dist-test-slave-czm5 krb5kdc[3279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:50.911751 3365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a23767e317b4e0e8572dd3c60efc41b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } }
I20260621 05:15:50.912572 3367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a23767e317b4e0e8572dd3c60efc41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a23767e317b4e0e8572dd3c60efc41b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } } }
I20260621 05:15:50.912657 3367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:50.912475 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a23767e317b4e0e8572dd3c60efc41b. Latest consensus state: current_term: 1 leader_uuid: "8a23767e317b4e0e8572dd3c60efc41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a23767e317b4e0e8572dd3c60efc41b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } } }
I20260621 05:15:50.912758 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b [sys.catalog]: This master's current role is: LEADER
W20260621 05:15:50.913044 3379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:15:50.913084 3379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:15:50.913184 3381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:50.913322 3381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:50.915199 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:50.905554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.0.98.62:42229)
0621 05:15:50.905754 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:50.905767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:50.905788 (+ 21us) server_negotiation.cc:407] Connection header received
0621 05:15:50.906224 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:50.906230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:50.906271 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:50.906420 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:50.907002 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:50.908002 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:50.908713 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:50.909072 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:50.909182 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:50.911669 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:50.911672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:50.911673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:50.911682 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:50.913580 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:50.913927 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:50.913929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:50.913930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:50.914014 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:50.914211 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:50.914212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:50.914212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:50.914537 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:50.914673 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:50.914884 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 05:15:50.914979 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260621 05:15:50.915712 3381 catalog_manager.cc:1347] Generated new cluster ID: afde136f217b430cbb67ec143e3edec4
I20260621 05:15:50.915804 3381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:50.916085 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:15:50.931613 3381 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:50.932376 3381 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:50.940402 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:50.936497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.0.98.62:42229)
0621 05:15:50.936533 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:50.936534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:50.936663 (+ 129us) server_negotiation.cc:407] Connection header received
0621 05:15:50.936736 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:50.936736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:50.936745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:50.936805 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:50.937216 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:50.937903 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:50.938938 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:50.939222 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:50.939275 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:50.939949 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:50.939950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:50.939950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:50.940156 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:50.940173 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:50.940178 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:50.940231 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:50.940306 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:15:50.940334 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:15:50.951421 3381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a23767e317b4e0e8572dd3c60efc41b: Generated new TSK 0
I20260621 05:15:50.951572 3381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:15:50.956892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 3292
2026-06-21T05:15:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1271 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:51 dist-test-slave-czm5 krb5kdc[3393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:51 dist-test-slave-czm5 krb5kdc[3393](info): set up 2 sockets
Jun 21 05:15:51 dist-test-slave-czm5 krb5kdc[3393](info): commencing operation
krb5kdc: starting...
W20260621 05:15:51.951257 392 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:52.056092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:52.131203 3406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:52.131443 3406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:52.131477 3406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:52.132614 3406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:52.132679 3406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:52.132702 3406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:52.132715 3406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:52.132727 3406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:52.134023 3406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:52.134181 3406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:52.134356 3406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:52.138895 3412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:52.138873 3411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:52.138958 3406 server_base.cc:1034] running on GCE node
W20260621 05:15:52.138882 3414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:52.139356 3406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:52.139683 3406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:52.140833 3406 hybrid_clock.cc:648] HybridClock initialized: now 1782018952140816 us; error 29 us; skew 500 ppm
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:52.143230 3406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:52.143587 3406 webserver.cc:460] Webserver started at http://127.0.98.62:33451/ using document root <none> and password file <none>
I20260621 05:15:52.143810 3406 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:52.143854 3406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:52.143936 3406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:52.144868 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "8130e64f76ab4d1d909432c4a80063c7"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "24a142e853566b8e4be945c2170172d4"
server_key_iv: "253cf38f4de16456814e9c2f15fbd7e7"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.145165 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "8130e64f76ab4d1d909432c4a80063c7"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "24a142e853566b8e4be945c2170172d4"
server_key_iv: "253cf38f4de16456814e9c2f15fbd7e7"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.146354 3406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:15:52.146993 3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.147138 3406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.147217 3406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "8130e64f76ab4d1d909432c4a80063c7"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "24a142e853566b8e4be945c2170172d4"
server_key_iv: "253cf38f4de16456814e9c2f15fbd7e7"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.147269 3406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:52.155802 3406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:52.157428 3406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:52.157527 3406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:52.161109 3406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41361
I20260621 05:15:52.161181 3473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41361 every 8 connection(s)
I20260621 05:15:52.161475 3406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:52.162348 3474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:52.164371 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7: Bootstrap starting.
I20260621 05:15:52.164924 3474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:52.165190 3474 log.cc:824] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:52.165838 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7: No bootstrap required, opened a new log
I20260621 05:15:52.167352 3474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130e64f76ab4d1d909432c4a80063c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41361 } }
I20260621 05:15:52.167508 3474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:52.167552 3474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8130e64f76ab4d1d909432c4a80063c7, State: Initialized, Role: FOLLOWER
I20260621 05:15:52.167722 3474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130e64f76ab4d1d909432c4a80063c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41361 } }
I20260621 05:15:52.167802 3474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:52.167845 3474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:52.167906 3474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:52.168650 3474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130e64f76ab4d1d909432c4a80063c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41361 } }
I20260621 05:15:52.168761 3474 leader_election.cc:302] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8130e64f76ab4d1d909432c4a80063c7; no voters:
I20260621 05:15:52.168936 3474 leader_election.cc:288] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:52.168960 3477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:52.169107 3477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [term 1 LEADER]: Becoming Leader. State: Replica: 8130e64f76ab4d1d909432c4a80063c7, State: Running, Role: LEADER
I20260621 05:15:52.169171 3474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:52.169238 3477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130e64f76ab4d1d909432c4a80063c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41361 } }
I20260621 05:15:52.169603 3479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8130e64f76ab4d1d909432c4a80063c7. Latest consensus state: current_term: 1 leader_uuid: "8130e64f76ab4d1d909432c4a80063c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130e64f76ab4d1d909432c4a80063c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41361 } } }
I20260621 05:15:52.169595 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8130e64f76ab4d1d909432c4a80063c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130e64f76ab4d1d909432c4a80063c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41361 } } }
I20260621 05:15:52.169751 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:52.169865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 3406
I20260621 05:15:52.169970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:52.169972 3483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:52.169970 3479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:52.170182 392 external_mini_cluster.cc:1428] Setting key 0e8b68c2797c41a461c36fe83d2b58fe
I20260621 05:15:52.170432 3483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:52.172374 3483 catalog_manager.cc:1347] Generated new cluster ID: 5ac7c5266ccb4dedb268d4c755a1741b
I20260621 05:15:52.172430 3483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:52.180945 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:52.170852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.0.98.62:41361)
0621 05:15:52.171766 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:52.171774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:52.171804 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:15:52.171917 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:52.171924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:52.171985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:52.172118 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:52.172832 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.173724 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.174486 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.174990 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.175110 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:52.177422 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:52.177426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:52.177426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:52.177441 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:52.179359 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.179794 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.179797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.179797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.179889 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.180092 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.180093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.180093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.180312 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:52.180478 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:52.180615 (+ 137us) server_negotiation.cc:299] Negotiation successful
0621 05:15:52.180759 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":70,"threads_started":1}
I20260621 05:15:52.188248 3483 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:52.188887 3483 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:52.193097 3483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8130e64f76ab4d1d909432c4a80063c7: Generated new TSK 0
I20260621 05:15:52.193243 3483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:15:52.219013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41361
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:52.294297 3500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:52.294499 3500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:52.294535 3500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:52.295706 3500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:52.295778 3500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:52.295822 3500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:52.297202 3500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:52.297438 3500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:52.297650 3500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:52.302171 3506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:52.302203 3500 server_base.cc:1034] running on GCE node
W20260621 05:15:52.302177 3505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:52.302177 3508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:52.302526 3500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:52.302740 3500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:52.303871 3500 hybrid_clock.cc:648] HybridClock initialized: now 1782018952303860 us; error 28 us; skew 500 ppm
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:52.306097 3500 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:52.306504 3500 webserver.cc:460] Webserver started at http://127.0.98.1:34155/ using document root <none> and password file <none>
I20260621 05:15:52.306669 3500 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:52.306708 3500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:52.306790 3500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:52.307742 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc4f11eecf484dcab4c7c042683810cf"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "b243ddc7fa4b5134d2e9968410c92236"
server_key_iv: "f7c073ade2756e33dfa5d9b77031f351"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.308029 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc4f11eecf484dcab4c7c042683810cf"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "b243ddc7fa4b5134d2e9968410c92236"
server_key_iv: "f7c073ade2756e33dfa5d9b77031f351"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.309156 3500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:52.309803 3515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.309974 3500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:52.310046 3500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "fc4f11eecf484dcab4c7c042683810cf"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "b243ddc7fa4b5134d2e9968410c92236"
server_key_iv: "f7c073ade2756e33dfa5d9b77031f351"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.310101 3500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:52.334128 3500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:52.335911 3500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:52.336010 3500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:52.336159 3500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:52.336390 3500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:52.336421 3500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.336439 3500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:52.336452 3500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.341696 3500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34343
I20260621 05:15:52.341769 3628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34343 every 8 connection(s)
I20260621 05:15:52.342033 3500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:15:52.343164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 3500
I20260621 05:15:52.343242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:52.343441 392 external_mini_cluster.cc:1428] Setting key 9869f7edd0617b1ef8c3bcae3ae3081c
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260621 05:15:52.351485 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:52.342679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32933 (local address 127.0.98.62:41361)
0621 05:15:52.342800 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:52.342803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:52.342959 (+ 156us) server_negotiation.cc:407] Connection header received
0621 05:15:52.343511 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:52.343513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:52.343525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:52.343580 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:52.344556 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.345061 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.346091 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.346299 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.346380 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:52.348794 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:52.348795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:52.348795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:52.348801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:52.350173 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.350549 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.350550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.350550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.350632 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.350838 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.350839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.350839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.350965 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:52.351032 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:52.351306 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 05:15:52.351399 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 05:15:52.351675 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:52.342634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41361 (local address 127.0.98.1:32933)
0621 05:15:52.342834 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:52.342874 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:52.343406 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:52.343683 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:52.343688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:52.344291 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:52.344452 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:52.344457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.345178 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.345180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:52.345992 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:52.345994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.346395 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.346396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:52.346521 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:52.346971 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:52.346978 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:52.348698 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:52.350275 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:52.350281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:52.350283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:52.350482 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:52.350681 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:52.350682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:52.350683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:52.350778 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:52.351037 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:52.351042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:52.351186 (+ 144us) client_negotiation.cc:769] Sending connection context
0621 05:15:52.351286 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 05:15:52.351445 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260621 05:15:52.352151 3629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41361
I20260621 05:15:52.352239 3629 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:52.352411 3629 heartbeater.cc:511] Master 127.0.98.62:41361 requested a full tablet report, sending...
I20260621 05:15:52.352872 3438 ts_manager.cc:194] Registered new tserver with Master: fc4f11eecf484dcab4c7c042683810cf (127.0.98.1:34343)
I20260621 05:15:52.353440 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:15:52.366291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41361
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:52.443591 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:52.443819 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:52.443842 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:52.444902 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:52.444938 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:52.444969 3634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:52.446267 3634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:52.446424 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:52.446578 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:52.451056 3639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:52.451162 3634 server_base.cc:1034] running on GCE node
W20260621 05:15:52.451081 3642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:52.451067 3640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:52.451457 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:52.451728 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:52.452859 3634 hybrid_clock.cc:648] HybridClock initialized: now 1782018952452847 us; error 30 us; skew 500 ppm
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:52.455147 3634 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:15:52.455574 3634 webserver.cc:460] Webserver started at http://127.0.98.2:36215/ using document root <none> and password file <none>
I20260621 05:15:52.455837 3634 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:52.455881 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:52.456025 3634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:52.456946 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "5fbb9840663f49b88b0cb9eac70641d1"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "0f20e7ab3f6af6ce4ff7f5f3843780bf"
server_key_iv: "68c4a8649d67d1d20f8d0176264c2877"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.457291 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5fbb9840663f49b88b0cb9eac70641d1"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "0f20e7ab3f6af6ce4ff7f5f3843780bf"
server_key_iv: "68c4a8649d67d1d20f8d0176264c2877"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.458523 3634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:52.459152 3649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.459304 3634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:52.459383 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "5fbb9840663f49b88b0cb9eac70641d1"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "0f20e7ab3f6af6ce4ff7f5f3843780bf"
server_key_iv: "68c4a8649d67d1d20f8d0176264c2877"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.459432 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:52.473027 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:52.475004 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:52.475106 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:52.475252 3634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:52.475565 3634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:52.475600 3634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.475656 3634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:52.475682 3634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.480947 3634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42009
I20260621 05:15:52.481016 3762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42009 every 8 connection(s)
I20260621 05:15:52.481277 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:52.488937 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:52.481894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42395 (local address 127.0.98.62:41361)
0621 05:15:52.481934 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:52.481936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:52.482094 (+ 158us) server_negotiation.cc:407] Connection header received
0621 05:15:52.482496 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:52.482497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:52.482506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:52.482567 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:52.483285 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.483832 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.484694 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.484863 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.484908 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:52.486765 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:52.486766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:52.486766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:52.486773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:52.487926 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.488248 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.488249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.488249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.488294 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.488487 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.488488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.488488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.488576 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:52.488636 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:52.488825 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 05:15:52.488880 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:15:52.489104 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:52.481857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41361 (local address 127.0.98.2:42395)
0621 05:15:52.481992 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:52.482027 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:52.482421 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:52.482608 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:52.482611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:52.483067 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:52.483218 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:52.483222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.483907 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.483908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:52.484628 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:52.484630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.484923 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.484923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:52.485006 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:52.485451 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:52.485459 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:52.486688 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:52.487986 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:52.487991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:52.487993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:52.488190 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:52.488344 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:52.488344 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:52.488345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:52.488434 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:52.488637 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:52.488640 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:52.488732 (+ 92us) client_negotiation.cc:769] Sending connection context
0621 05:15:52.488804 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:15:52.488901 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260621 05:15:52.489526 3763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41361
I20260621 05:15:52.489622 3763 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:52.489775 3763 heartbeater.cc:511] Master 127.0.98.62:41361 requested a full tablet report, sending...
I20260621 05:15:52.490106 3438 ts_manager.cc:194] Registered new tserver with Master: 5fbb9840663f49b88b0cb9eac70641d1 (127.0.98.2:42009)
I20260621 05:15:52.490113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 3634
I20260621 05:15:52.490211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:52.490376 392 external_mini_cluster.cc:1428] Setting key 250acd811540dce465dddfd9ae1daa95
I20260621 05:15:52.490521 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42395
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:15:52.512233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41361
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:52.588088 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:52.588248 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:52.588268 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:52.589277 3768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:52.589308 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:52.589332 3768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:52.590636 3768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:52.590793 3768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:52.590946 3768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:52.595557 3773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:52.595554 3776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:52.595732 3768 server_base.cc:1034] running on GCE node
W20260621 05:15:52.595583 3774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:52.596051 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:52.596287 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:52.597436 3768 hybrid_clock.cc:648] HybridClock initialized: now 1782018952597418 us; error 33 us; skew 500 ppm
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:52.599860 3768 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:15:52.600283 3768 webserver.cc:460] Webserver started at http://127.0.98.3:44629/ using document root <none> and password file <none>
I20260621 05:15:52.600508 3768 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:52.600569 3768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:52.600692 3768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:52.601625 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "5aa3987c0f15475ca3b97d9f2de92828"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "da836b031678315094798161c6857033"
server_key_iv: "fe9aac1f55bac1d07a8f52e511f5a356"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.601981 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5aa3987c0f15475ca3b97d9f2de92828"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "da836b031678315094798161c6857033"
server_key_iv: "fe9aac1f55bac1d07a8f52e511f5a356"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.603277 3768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:52.604012 3783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.604166 3768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.604255 3768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "5aa3987c0f15475ca3b97d9f2de92828"
format_stamp: "Formatted at 2026-06-21 05:15:52 on dist-test-slave-czm5"
server_key: "da836b031678315094798161c6857033"
server_key_iv: "fe9aac1f55bac1d07a8f52e511f5a356"
server_key_version: "encryptionkey@0"
I20260621 05:15:52.604337 3768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:52.614620 3768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:52.616780 3768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:52.616928 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:52.617137 3768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:52.617426 3768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:52.617501 3768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.617539 3768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:52.617568 3768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:52.623168 3768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42731
I20260621 05:15:52.623208 3896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42731 every 8 connection(s)
I20260621 05:15:52.623492 3768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:15:52.626288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 3768
I20260621 05:15:52.626389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:52.626574 392 external_mini_cluster.cc:1428] Setting key f0a941293c521b7abe53ab4becaf5a19
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:52.632153 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:52.624135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38203 (local address 127.0.98.62:41361)
0621 05:15:52.624175 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:52.624176 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:52.624354 (+ 178us) server_negotiation.cc:407] Connection header received
0621 05:15:52.624803 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:52.624804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:52.624814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:52.624864 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:52.625672 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.626182 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.627540 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.627816 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.627882 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:52.629673 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:52.629674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:52.629675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:52.629682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:52.630806 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.631210 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.631211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.631212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.631266 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.631510 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.631511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.631511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.631661 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:52.631722 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:52.631971 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 05:15:52.632048 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:15:52.632333 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:52.624131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41361 (local address 127.0.98.3:38203)
0621 05:15:52.624261 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:52.624292 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:52.624725 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:52.624913 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:52.624916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:52.625428 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:52.625608 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:52.625613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.626279 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.626280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:52.627456 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:52.627458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.627899 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.627900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:52.628033 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:52.628374 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:52.628381 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:52.629587 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:52.630916 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:52.630922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:52.630925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:52.631137 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:52.631335 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:52.631335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:52.631336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:52.631437 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:52.631714 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:52.631720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:52.631847 (+ 127us) client_negotiation.cc:769] Sending connection context
0621 05:15:52.631949 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 05:15:52.632106 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260621 05:15:52.632763 3897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41361
I20260621 05:15:52.632844 3897 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:52.633008 3897 heartbeater.cc:511] Master 127.0.98.62:41361 requested a full tablet report, sending...
I20260621 05:15:52.633318 3438 ts_manager.cc:194] Registered new tserver with Master: 5aa3987c0f15475ca3b97d9f2de92828 (127.0.98.3:42731)
I20260621 05:15:52.633694 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38203
I20260621 05:15:52.638125 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:52.658707 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:52.652417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.0.98.62:41361)
0621 05:15:52.652467 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:52.652468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:52.652592 (+ 124us) server_negotiation.cc:407] Connection header received
0621 05:15:52.652631 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:52.652632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:52.652640 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:52.652686 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:52.653177 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.653621 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.654279 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:52.654467 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:52.654510 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:52.656537 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:52.656538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:52.656538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:52.656544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:52.657826 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.658088 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.658088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.658088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.658139 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:52.658333 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:52.658333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:52.658333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:52.658449 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:52.658514 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:52.658601 (+ 87us) server_negotiation.cc:299] Negotiation successful
0621 05:15:52.658652 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260621 05:15:52.658915 3438 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34254
I20260621 05:15:52.659366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 3500
I20260621 05:15:52.663700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 3634
I20260621 05:15:52.667919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 3768
I20260621 05:15:52.671960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 3406
2026-06-21T05:15:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1716 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3909](info): set up 2 sockets
Jun 21 05:15:52 dist-test-slave-czm5 krb5kdc[3909](info): commencing operation
krb5kdc: starting...
W20260621 05:15:53.667208 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:53 dist-test-slave-czm5 krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:53.773928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:53.849888 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:53.850122 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:53.850157 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:53.851259 3922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:53.851320 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:53.851339 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:53.851359 3922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:53.851382 3922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:53.852839 3922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:53.853080 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:53.853292 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:53.857753 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:53.857759 3927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:53.857894 3922 server_base.cc:1034] running on GCE node
W20260621 05:15:53.857801 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:53.858222 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:53.858474 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:53.859652 3922 hybrid_clock.cc:648] HybridClock initialized: now 1782018953859610 us; error 28 us; skew 500 ppm
Jun 21 05:15:53 dist-test-slave-czm5 krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:53.861934 3922 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:53.862349 3922 webserver.cc:460] Webserver started at http://127.0.98.62:39161/ using document root <none> and password file <none>
I20260621 05:15:53.862555 3922 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:53.862597 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:53.862746 3922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:53.863780 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb"
format_stamp: "Formatted at 2026-06-21 05:15:53 on dist-test-slave-czm5"
server_key: "3ef74b9c700af036a987a590fa500cb3"
server_key_iv: "3db1fdcacc2ceeaad62eea086dcfb4cd"
server_key_version: "encryptionkey@0"
I20260621 05:15:53.864106 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb"
format_stamp: "Formatted at 2026-06-21 05:15:53 on dist-test-slave-czm5"
server_key: "3ef74b9c700af036a987a590fa500cb3"
server_key_iv: "3db1fdcacc2ceeaad62eea086dcfb4cd"
server_key_version: "encryptionkey@0"
I20260621 05:15:53.865291 3922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:15:53.865931 3937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:53.866089 3922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:53.866179 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb"
format_stamp: "Formatted at 2026-06-21 05:15:53 on dist-test-slave-czm5"
server_key: "3ef74b9c700af036a987a590fa500cb3"
server_key_iv: "3db1fdcacc2ceeaad62eea086dcfb4cd"
server_key_version: "encryptionkey@0"
I20260621 05:15:53.866250 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:53.887727 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:53.889456 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:53.889609 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:53.893365 3922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46011
I20260621 05:15:53.893397 3989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46011 every 8 connection(s)
I20260621 05:15:53.893723 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:53.894533 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:53.896975 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb: Bootstrap starting.
I20260621 05:15:53.897564 3990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:53.897848 3990 log.cc:824] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:53.898152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 3922
I20260621 05:15:53.898233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:53.898418 392 external_mini_cluster.cc:1428] Setting key 14dd61b65a20da1c83ad8fbad07a2699
I20260621 05:15:53.898593 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb: No bootstrap required, opened a new log
I20260621 05:15:53.900372 3990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46011 } }
I20260621 05:15:53.900521 3990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:53.900590 3990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5ce7369ff4b39b2043e5b3b5ea3cb, State: Initialized, Role: FOLLOWER
I20260621 05:15:53.900736 3990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46011 } }
I20260621 05:15:53.900830 3990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:53.900867 3990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:53.900916 3990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:53.901844 3990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46011 } }
I20260621 05:15:53.901964 3990 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa5ce7369ff4b39b2043e5b3b5ea3cb; no voters:
I20260621 05:15:53.902136 3990 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:53.902156 3995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:53.902280 3995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [term 1 LEADER]: Becoming Leader. State: Replica: 5aa5ce7369ff4b39b2043e5b3b5ea3cb, State: Running, Role: LEADER
I20260621 05:15:53.902418 3990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:53.902424 3995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46011 } }
I20260621 05:15:53.903916 3996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46011 } } }
I20260621 05:15:53.903924 3997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa5ce7369ff4b39b2043e5b3b5ea3cb. Latest consensus state: current_term: 1 leader_uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa5ce7369ff4b39b2043e5b3b5ea3cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46011 } } }
I20260621 05:15:53.904034 3996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:53.904064 3997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:53.904217 4009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:53.904819 4009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:15:53 dist-test-slave-czm5 krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:53.906735 4009 catalog_manager.cc:1347] Generated new cluster ID: 214900294ddb4e6c946ce76ae941d02d
I20260621 05:15:53.906787 4009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:53.909993 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:53.898896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.0.98.62:46011)
0621 05:15:53.899055 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:53.899063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:53.899093 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:15:53.900358 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:53.900365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:53.900415 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:53.900539 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:53.901335 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:53.902359 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:53.903327 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:53.903985 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:53.904089 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:53.906817 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:53.906821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:53.906821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:53.906832 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:53.908370 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:53.908727 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:53.908730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:53.908730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:53.908827 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:53.909039 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:53.909040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:53.909040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:53.909258 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:53.909438 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:53.909525 (+ 87us) server_negotiation.cc:299] Negotiation successful
0621 05:15:53.909657 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260621 05:15:53.929467 4009 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:53.929988 4009 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:53.937866 4009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa5ce7369ff4b39b2043e5b3b5ea3cb: Generated new TSK 0
I20260621 05:15:53.938048 4009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:15:53.964514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46011
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:54.041476 4016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:54.041682 4016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:54.041719 4016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:54.042833 4016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:54.042896 4016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:54.042939 4016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:54.044376 4016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:54.044634 4016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:54.044850 4016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:54.049628 4021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:54.049644 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:54.049680 4016 server_base.cc:1034] running on GCE node
W20260621 05:15:54.049633 4024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:54.049965 4016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:54.050187 4016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:54.051329 4016 hybrid_clock.cc:648] HybridClock initialized: now 1782018954051312 us; error 28 us; skew 500 ppm
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:54.053619 4016 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:54.054021 4016 webserver.cc:460] Webserver started at http://127.0.98.1:37135/ using document root <none> and password file <none>
I20260621 05:15:54.054224 4016 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:54.054318 4016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:54.054468 4016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:54.055426 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "fbb724e6dcc549cdb9b69e3dc9ffe588"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "c98933cc5e8c272bcf72a791c31b1515"
server_key_iv: "1ba6ad0cb8687d07ed13887c3069649b"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.055789 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fbb724e6dcc549cdb9b69e3dc9ffe588"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "c98933cc5e8c272bcf72a791c31b1515"
server_key_iv: "1ba6ad0cb8687d07ed13887c3069649b"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.057065 4016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:15:54.057698 4031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.057839 4016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:54.057930 4016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "fbb724e6dcc549cdb9b69e3dc9ffe588"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "c98933cc5e8c272bcf72a791c31b1515"
server_key_iv: "1ba6ad0cb8687d07ed13887c3069649b"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.058009 4016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:54.096671 4016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:54.099301 4016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:54.099452 4016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:54.099722 4016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:54.100071 4016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:54.100131 4016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.100183 4016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:54.100216 4016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.107301 4016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44933
I20260621 05:15:54.107338 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44933 every 8 connection(s)
I20260621 05:15:54.107714 4016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:15:54.108861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 4016
I20260621 05:15:54.108949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:54.109175 392 external_mini_cluster.cc:1428] Setting key e3a319e674a60d01e5588dbbe9313f3f
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260621 05:15:54.118353 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:54.108467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44483 (local address 127.0.98.62:46011)
0621 05:15:54.108510 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:54.108512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:54.108719 (+ 207us) server_negotiation.cc:407] Connection header received
0621 05:15:54.109817 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:54.109818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:54.109830 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:54.109889 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:54.110481 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.111334 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.112649 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.113070 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.113147 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:54.115377 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:54.115378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:54.115379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:54.115388 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:54.116873 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.117320 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.117321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.117321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.117377 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.117661 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.117662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.117663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.117781 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:54.117854 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:54.118163 (+ 309us) server_negotiation.cc:299] Negotiation successful
0621 05:15:54.118248 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:15:54.118526 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:54.108461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46011 (local address 127.0.98.1:44483)
0621 05:15:54.108603 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:54.108638 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:54.109716 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:54.109957 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:54.109960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:54.110209 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:54.110394 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:54.110398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.111420 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.111421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:54.112573 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:54.112576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.113149 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.113150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:54.113242 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:54.113763 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:54.113771 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:54.115279 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:54.116996 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:54.117002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:54.117005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:54.117235 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:54.117454 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:54.117455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:54.117455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:54.117595 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:54.117845 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:54.117851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:54.118008 (+ 157us) client_negotiation.cc:769] Sending connection context
0621 05:15:54.118144 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 05:15:54.118294 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260621 05:15:54.119004 4145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46011
I20260621 05:15:54.119094 4145 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:54.119232 4145 heartbeater.cc:511] Master 127.0.98.62:46011 requested a full tablet report, sending...
I20260621 05:15:54.119757 3954 ts_manager.cc:194] Registered new tserver with Master: fbb724e6dcc549cdb9b69e3dc9ffe588 (127.0.98.1:44933)
I20260621 05:15:54.120656 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260621 05:15:54.427078 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.317s user 0.001s sys 0.002s
I20260621 05:15:54.427493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46011
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:54.508526 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:54.508716 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:54.508736 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:54.509781 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:54.509819 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:54.509862 4150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:54.511229 4150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:54.511451 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:54.511668 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:54.516413 4155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:54.516381 4158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:54.516386 4156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:54.516673 4150 server_base.cc:1034] running on GCE node
I20260621 05:15:54.516870 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:54.517094 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:54.518246 4150 hybrid_clock.cc:648] HybridClock initialized: now 1782018954518237 us; error 36 us; skew 500 ppm
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:54.520547 4150 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:15:54.520932 4150 webserver.cc:460] Webserver started at http://127.0.98.2:36309/ using document root <none> and password file <none>
I20260621 05:15:54.521164 4150 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:54.521245 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:54.521376 4150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:54.522337 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "4be22c098e024db8bc0c1ea675b805a4"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "84ef5d8e3df33727ba9d320659216013"
server_key_iv: "85618c86a108d9a97f56153a2d34748f"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.522670 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4be22c098e024db8bc0c1ea675b805a4"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "84ef5d8e3df33727ba9d320659216013"
server_key_iv: "85618c86a108d9a97f56153a2d34748f"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.523891 4150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:54.524501 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.524675 4150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:54.524757 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "4be22c098e024db8bc0c1ea675b805a4"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "84ef5d8e3df33727ba9d320659216013"
server_key_iv: "85618c86a108d9a97f56153a2d34748f"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.524822 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:54.539809 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:54.541455 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:54.541610 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:54.541838 4150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:54.542105 4150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:54.542153 4150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.542191 4150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:54.542215 4150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.547514 4150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32897
I20260621 05:15:54.547581 4278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32897 every 8 connection(s)
I20260621 05:15:54.547868 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:15:54.552511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 4150
I20260621 05:15:54.552620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:54.552809 392 external_mini_cluster.cc:1428] Setting key aec577a417d91d0d90b7182c730b4a39
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:54.556568 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:54.548468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46011 (local address 127.0.98.2:34985)
0621 05:15:54.548605 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:54.548643 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:54.549474 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:54.549674 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:54.549676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:54.549897 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:54.550063 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:54.550067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.550688 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.550689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:54.551416 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:54.551417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.551734 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.551735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:54.551825 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:54.552193 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:54.552200 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:54.553670 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:54.555169 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:54.555175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:54.555177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:54.555485 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:54.555698 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:54.555699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:54.555699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:54.555826 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:54.556116 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:54.556121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:54.556223 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 05:15:54.556306 (+ 83us) client_negotiation.cc:240] Negotiation successful
0621 05:15:54.556415 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260621 05:15:54.556571 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:54.548506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34985 (local address 127.0.98.62:46011)
0621 05:15:54.548548 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:54.548550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:54.548722 (+ 172us) server_negotiation.cc:407] Connection header received
0621 05:15:54.549564 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:54.549565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:54.549573 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:54.549625 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:54.550128 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.550619 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.551484 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.551670 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.551712 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:54.553765 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:54.553766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:54.553766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:54.553772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:54.555094 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.555566 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.555567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.555567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.555609 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.555891 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.555892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.555892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.556052 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:54.556119 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:54.556347 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 05:15:54.556435 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:15:54.557063 4279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46011
I20260621 05:15:54.557145 4279 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:54.557328 4279 heartbeater.cc:511] Master 127.0.98.62:46011 requested a full tablet report, sending...
I20260621 05:15:54.557708 3954 ts_manager.cc:194] Registered new tserver with Master: 4be22c098e024db8bc0c1ea675b805a4 (127.0.98.2:32897)
I20260621 05:15:54.558051 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34985
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:15:54.576084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46011
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:54.653857 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:54.654038 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:54.654059 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:54.655211 4284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:54.655248 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:54.655299 4284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:54.656754 4284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:54.656968 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:54.657179 4284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:54.661768 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:54.661788 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:54.661779 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:54.661988 4284 server_base.cc:1034] running on GCE node
I20260621 05:15:54.662187 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:54.662443 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:54.663589 4284 hybrid_clock.cc:648] HybridClock initialized: now 1782018954663589 us; error 38 us; skew 500 ppm
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:54.665889 4284 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:15:54.666343 4284 webserver.cc:460] Webserver started at http://127.0.98.3:46605/ using document root <none> and password file <none>
I20260621 05:15:54.666553 4284 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:54.666615 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:54.666759 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:54.667778 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ad876e21b1a4a7daebefcce0fb7fafe"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "a99c784e2ec28e04dd7906e623fb2e5a"
server_key_iv: "01125430f48beabc62c93813cfe67728"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.668107 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ad876e21b1a4a7daebefcce0fb7fafe"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "a99c784e2ec28e04dd7906e623fb2e5a"
server_key_iv: "01125430f48beabc62c93813cfe67728"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.669911 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260621 05:15:54.670560 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.670719 4284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.670814 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "6ad876e21b1a4a7daebefcce0fb7fafe"
format_stamp: "Formatted at 2026-06-21 05:15:54 on dist-test-slave-czm5"
server_key: "a99c784e2ec28e04dd7906e623fb2e5a"
server_key_iv: "01125430f48beabc62c93813cfe67728"
server_key_version: "encryptionkey@0"
I20260621 05:15:54.670889 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:54.682132 4284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:54.684224 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:54.684384 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:54.684595 4284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:54.684871 4284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:54.684918 4284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.684959 4284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:54.684988 4284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:54.690608 4284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38025
I20260621 05:15:54.690649 4412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38025 every 8 connection(s)
I20260621 05:15:54.690943 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:54.699563 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:54.691586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41453 (local address 127.0.98.62:46011)
0621 05:15:54.691676 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:54.691678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:54.691861 (+ 183us) server_negotiation.cc:407] Connection header received
0621 05:15:54.692759 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:54.692760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:54.692772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:54.692822 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:54.693373 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.693863 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.694866 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.695059 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.695127 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:54.697118 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:54.697119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:54.697120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:54.697127 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:54.698512 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.698837 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.698838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.698838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.698884 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.699077 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.699078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.699078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.699173 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:54.699221 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:54.699447 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 05:15:54.699502 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 05:15:54.699729 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:54.691561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46011 (local address 127.0.98.3:41453)
0621 05:15:54.691727 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:54.691769 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:54.692649 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:54.692885 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:54.692889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:54.693126 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:54.693288 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:54.693293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.693952 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.693953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:54.694756 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:54.694758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.695144 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.695145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:54.695261 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:54.695739 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:54.695747 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:54.697038 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:54.698577 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:54.698581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:54.698582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:54.698779 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:54.698937 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:54.698938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:54.698938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:54.699028 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:54.699221 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:54.699225 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:54.699326 (+ 101us) client_negotiation.cc:769] Sending connection context
0621 05:15:54.699403 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 05:15:54.699522 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260621 05:15:54.700098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 4284
I20260621 05:15:54.700215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:54.700235 4413 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46011
I20260621 05:15:54.700379 4413 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:54.700452 392 external_mini_cluster.cc:1428] Setting key 83b6526404e8a42ef7532ccc09d10470
I20260621 05:15:54.700569 4413 heartbeater.cc:511] Master 127.0.98.62:46011 requested a full tablet report, sending...
I20260621 05:15:54.700922 3954 ts_manager.cc:194] Registered new tserver with Master: 6ad876e21b1a4a7daebefcce0fb7fafe (127.0.98.3:38025)
I20260621 05:15:54.701327 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:15:54.701380 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41453
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:15:54.722467 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:54.715184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.0.98.1:44933)
0621 05:15:54.715298 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:54.715304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:54.715325 (+ 21us) server_negotiation.cc:407] Connection header received
0621 05:15:54.715363 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:54.715365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:54.715380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:54.715461 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:54.716061 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.716746 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.717487 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.717770 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.717849 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:54.719980 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:54.719987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:54.719987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:54.719993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:54.721461 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.721872 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.721874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.721874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.721927 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.722072 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.722073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.722073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.722182 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:54.722283 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:54.722334 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 05:15:54.722391 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:54.729646 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:54.723674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.0.98.62:46011)
0621 05:15:54.723728 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:54.723730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:54.723761 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:15:54.723810 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:54.723811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:54.723819 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:54.723867 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:54.724287 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.724740 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.725349 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:54.725515 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:54.725559 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:54.727473 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:54.727475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:54.727475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:54.727481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:54.728829 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.729110 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.729110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.729111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.729153 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:54.729283 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:54.729283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:54.729283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:54.729396 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:54.729465 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:54.729535 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 05:15:54.729589 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260621 05:15:54.729878 3954 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55480
I20260621 05:15:54.730238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 4016
I20260621 05:15:54.734488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 4150
I20260621 05:15:54.738322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 4284
I20260621 05:15:54.742372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 3922
2026-06-21T05:15:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2069 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[4425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[4425](info): set up 2 sockets
Jun 21 05:15:54 dist-test-slave-czm5 krb5kdc[4425](info): commencing operation
krb5kdc: starting...
W20260621 05:15:55.735387 392 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:55 dist-test-slave-czm5 krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:55.846048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39617
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:55.923836 4438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:55.924079 4438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:55.924116 4438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:55.925154 4438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:55.925213 4438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:55.925235 4438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:55.925256 4438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:55.925278 4438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:55.926692 4438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:55.926952 4438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:55.927165 4438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:55.931480 4443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:55.931547 4444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:55.931499 4446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:55.931515 4438 server_base.cc:1034] running on GCE node
I20260621 05:15:55.931954 4438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:55.932225 4438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:55.933377 4438 hybrid_clock.cc:648] HybridClock initialized: now 1782018955933371 us; error 33 us; skew 500 ppm
Jun 21 05:15:55 dist-test-slave-czm5 krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:55.939814 4438 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:55.940122 4438 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:55.940187 4438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:55.940318 4438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:55.941273 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "5c662161941b40d28eddd6a7e57db310"
format_stamp: "Formatted at 2026-06-21 05:15:55 on dist-test-slave-czm5"
server_key: "d33d93247711c7154f4034ec0bc56b3d"
server_key_iv: "a8552060f7e9d7c8e6c4aee0ede910e2"
server_key_version: "encryptionkey@0"
I20260621 05:15:55.941702 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c662161941b40d28eddd6a7e57db310"
format_stamp: "Formatted at 2026-06-21 05:15:55 on dist-test-slave-czm5"
server_key: "d33d93247711c7154f4034ec0bc56b3d"
server_key_iv: "a8552060f7e9d7c8e6c4aee0ede910e2"
server_key_version: "encryptionkey@0"
I20260621 05:15:55.942894 4438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:15:55.943600 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:55.943800 4438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:55.943890 4438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "5c662161941b40d28eddd6a7e57db310"
format_stamp: "Formatted at 2026-06-21 05:15:55 on dist-test-slave-czm5"
server_key: "d33d93247711c7154f4034ec0bc56b3d"
server_key_iv: "a8552060f7e9d7c8e6c4aee0ede910e2"
server_key_version: "encryptionkey@0"
I20260621 05:15:55.943975 4438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:55.957492 4438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:55.959606 4438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:55.959829 4438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:55.963338 4438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39617
I20260621 05:15:55.963387 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39617 every 8 connection(s)
I20260621 05:15:55.963655 4438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:55.964532 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:55.966744 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310: Bootstrap starting.
I20260621 05:15:55.967336 4505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:55.967669 4505 log.cc:824] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:55.968268 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310: No bootstrap required, opened a new log
I20260621 05:15:55.969789 4505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c662161941b40d28eddd6a7e57db310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } }
I20260621 05:15:55.969923 4505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:55.969995 4505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c662161941b40d28eddd6a7e57db310, State: Initialized, Role: FOLLOWER
I20260621 05:15:55.970175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 4438
I20260621 05:15:55.970157 4505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c662161941b40d28eddd6a7e57db310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } }
I20260621 05:15:55.970232 4505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:55.970273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:55.970286 4505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:55.970326 4505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:55.970472 392 external_mini_cluster.cc:1428] Setting key f917b90e5d3bed3f656a1ec621ef4117
I20260621 05:15:55.971379 4505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c662161941b40d28eddd6a7e57db310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } }
I20260621 05:15:55.971508 4505 leader_election.cc:302] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c662161941b40d28eddd6a7e57db310; no voters:
I20260621 05:15:55.971705 4505 leader_election.cc:288] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:55.971727 4510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:55.971887 4510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [term 1 LEADER]: Becoming Leader. State: Replica: 5c662161941b40d28eddd6a7e57db310, State: Running, Role: LEADER
I20260621 05:15:55.971966 4505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:55.972005 4510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c662161941b40d28eddd6a7e57db310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } }
I20260621 05:15:55.972499 4511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c662161941b40d28eddd6a7e57db310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c662161941b40d28eddd6a7e57db310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } } }
I20260621 05:15:55.972568 4511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:55.972766 4512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c662161941b40d28eddd6a7e57db310. Latest consensus state: current_term: 1 leader_uuid: "5c662161941b40d28eddd6a7e57db310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c662161941b40d28eddd6a7e57db310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } } }
I20260621 05:15:55.973207 4512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310 [sys.catalog]: This master's current role is: LEADER
W20260621 05:15:55.973824 4526 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:15:55.973992 4526 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:15:55.974064 4517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:55.974221 4517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:55.975797 4517 catalog_manager.cc:1347] Generated new cluster ID: 6361b77a34a4401e8abaf969ace7519a
I20260621 05:15:55.975862 4517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:15:55 dist-test-slave-czm5 krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:55.981806 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:55.970978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.0.98.62:39617)
0621 05:15:55.971176 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:55.971189 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:55.971213 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:15:55.971299 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:55.971303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:55.971342 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:55.971452 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:55.972322 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:55.973720 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:55.974802 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:55.975409 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:55.975536 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:55.978191 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:55.978196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:55.978196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:55.978210 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:55.980253 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:55.980613 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:55.980617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:55.980617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:55.980737 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:55.980928 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:55.980929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:55.980929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:55.981230 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:55.981385 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:55.981461 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 05:15:55.981610 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260621 05:15:55.987443 4517 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:55.987967 4517 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:55.998421 4517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310: Generated new TSK 0
I20260621 05:15:55.998595 4517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:15:56.022156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39617
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:56.112020 4531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:56.112205 4531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:56.112226 4531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:56.113306 4531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:56.113353 4531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:56.113379 4531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:56.114857 4531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:56.115091 4531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:56.115307 4531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:56.119902 4536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.119993 4539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.119899 4537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:56.120241 4531 server_base.cc:1034] running on GCE node
I20260621 05:15:56.120448 4531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:56.120690 4531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:56.121850 4531 hybrid_clock.cc:648] HybridClock initialized: now 1782018956121837 us; error 27 us; skew 500 ppm
Jun 21 05:15:56 dist-test-slave-czm5 krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:56.124109 4531 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:56.124384 4531 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:56.124425 4531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:56.124503 4531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:56.125428 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "fcc0746b2a9c4d23bce9e36d6a823173"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "279cf6b5d654697c27c81e760306cbd7"
server_key_iv: "479afc2bdf5560141afff93758630b67"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.125752 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fcc0746b2a9c4d23bce9e36d6a823173"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "279cf6b5d654697c27c81e760306cbd7"
server_key_iv: "479afc2bdf5560141afff93758630b67"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.126976 4531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:15:56.127679 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.127837 4531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.127912 4531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "fcc0746b2a9c4d23bce9e36d6a823173"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "279cf6b5d654697c27c81e760306cbd7"
server_key_iv: "479afc2bdf5560141afff93758630b67"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.127976 4531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:56.152370 4531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:56.154102 4531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:56.154270 4531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:56.154503 4531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:56.154824 4531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:56.154858 4531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.154881 4531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:56.154925 4531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.160386 4531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34389
I20260621 05:15:56.160475 4658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34389 every 8 connection(s)
I20260621 05:15:56.160697 4531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:15:56 dist-test-slave-czm5 krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:56.166714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 4531
I20260621 05:15:56.166843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:56.167047 392 external_mini_cluster.cc:1428] Setting key 0db6dc9ffc7e43560de2345c292ce1fd
I20260621 05:15:56.169523 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.161378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58471 (local address 127.0.98.62:39617)
0621 05:15:56.161433 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.161435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.161624 (+ 189us) server_negotiation.cc:407] Connection header received
0621 05:15:56.162066 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.162067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.162081 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.162127 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:56.163001 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.163488 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.164491 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.164682 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.164733 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.166672 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:56.166674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:56.166674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:56.166682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:56.168150 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:56.168531 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:56.168532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:56.168533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:56.168584 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:56.168859 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:56.168861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:56.168861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:56.168978 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:56.169037 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.169384 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.169460 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:15:56.169735 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.161350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39617 (local address 127.0.98.1:58471)
0621 05:15:56.161510 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.161544 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.161986 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.162199 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.162202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.162431 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:56.162928 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.162932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.163560 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.163560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.164401 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.164403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.164748 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.164749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.164853 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.165249 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:56.165269 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:56.166585 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:56.168230 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:56.168236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:56.168239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:56.168451 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:56.168642 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:56.168643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:56.168644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:56.168779 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:56.169051 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:56.169056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:56.169212 (+ 156us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.169331 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.169489 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260621 05:15:56.170156 4659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39617
I20260621 05:15:56.170239 4659 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:56.170388 4659 heartbeater.cc:511] Master 127.0.98.62:39617 requested a full tablet report, sending...
I20260621 05:15:56.170894 4469 ts_manager.cc:194] Registered new tserver with Master: fcc0746b2a9c4d23bce9e36d6a823173 (127.0.98.1:34389)
I20260621 05:15:56.171820 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58471
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:15:56.190255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39617
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:56.267272 4664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:56.267509 4664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:56.267545 4664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:56.268656 4664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:56.268719 4664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:56.268761 4664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:56.270147 4664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:56.270396 4664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:56.270612 4664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:56.275424 4669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.275475 4672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.275419 4670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:56.275750 4664 server_base.cc:1034] running on GCE node
I20260621 05:15:56.275964 4664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:56.276196 4664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:56.277352 4664 hybrid_clock.cc:648] HybridClock initialized: now 1782018956277335 us; error 27 us; skew 500 ppm
Jun 21 05:15:56 dist-test-slave-czm5 krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:56.279587 4664 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:15:56.279919 4664 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:56.279999 4664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:56.280133 4664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:56.281111 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "d5c33d1cdb2846b19816e980a2bb80d3"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "cec10c274427c4848e437e4eb74980d8"
server_key_iv: "c328b440e6e2fce8fcff81c9761d8a3d"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.281457 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d5c33d1cdb2846b19816e980a2bb80d3"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "cec10c274427c4848e437e4eb74980d8"
server_key_iv: "c328b440e6e2fce8fcff81c9761d8a3d"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.282708 4664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:56.283319 4678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.283473 4664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:56.283561 4664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "d5c33d1cdb2846b19816e980a2bb80d3"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "cec10c274427c4848e437e4eb74980d8"
server_key_iv: "c328b440e6e2fce8fcff81c9761d8a3d"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.283669 4664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:56.296253 4664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:56.297946 4664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:56.298090 4664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:56.298287 4664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:56.298623 4664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:56.298671 4664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.298702 4664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:56.298728 4664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.304154 4664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41075
I20260621 05:15:56.304193 4791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41075 every 8 connection(s)
I20260621 05:15:56.304448 4664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:15:56 dist-test-slave-czm5 krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:56.312566 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.305000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58285 (local address 127.0.98.62:39617)
0621 05:15:56.305041 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.305043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.305243 (+ 200us) server_negotiation.cc:407] Connection header received
0621 05:15:56.305714 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.305715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.305724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.305770 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:56.306570 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.307053 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.308048 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.308233 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.308297 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.310109 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:56.310111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:56.310111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:56.310117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:56.311438 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:56.311759 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:56.311760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:56.311760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:56.311808 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:56.312014 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:56.312015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:56.312015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:56.312100 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:56.312144 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.312407 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.312490 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:15:56.312702 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.304994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39617 (local address 127.0.98.2:58285)
0621 05:15:56.305131 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.305161 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.305631 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.305824 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.305827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.306057 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:56.306501 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.306506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.307125 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.307126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.307979 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.307981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.308329 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.308329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.308416 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.308770 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:56.308777 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:56.310021 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:56.311502 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:56.311506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:56.311507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:56.311710 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:56.311862 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:56.311862 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:56.311862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:56.311969 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:56.312145 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:56.312148 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:56.312274 (+ 126us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.312381 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.312502 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260621 05:15:56.313124 4792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39617
I20260621 05:15:56.313217 4792 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:56.313360 4792 heartbeater.cc:511] Master 127.0.98.62:39617 requested a full tablet report, sending...
I20260621 05:15:56.313726 4469 ts_manager.cc:194] Registered new tserver with Master: d5c33d1cdb2846b19816e980a2bb80d3 (127.0.98.2:41075)
I20260621 05:15:56.314075 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58285
I20260621 05:15:56.314359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 4664
I20260621 05:15:56.314471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:56.314666 392 external_mini_cluster.cc:1428] Setting key e4eb260d6e0deeaea46954649d63aaf2
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:15:56.336673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39617
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:56.413544 4797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:56.413724 4797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:56.413746 4797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:56.414862 4797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:56.414903 4797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:56.414929 4797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:56.416419 4797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:56.416635 4797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:56.416841 4797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:56.421703 4803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.421681 4805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.421680 4802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:56.422164 4797 server_base.cc:1034] running on GCE node
I20260621 05:15:56.422372 4797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:56.422623 4797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:56.423774 4797 hybrid_clock.cc:648] HybridClock initialized: now 1782018956423749 us; error 39 us; skew 500 ppm
Jun 21 05:15:56 dist-test-slave-czm5 krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:56.426669 4797 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:15:56.426919 4797 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:56.426959 4797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:56.427047 4797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:56.428040 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "9726c26dd8894fdfb859691ee6cafef3"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "a0fdf79f2d1e397dfd53439aa9b55f39"
server_key_iv: "314dcde0aa96f57cc5581a98f6c103ff"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.428315 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9726c26dd8894fdfb859691ee6cafef3"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "a0fdf79f2d1e397dfd53439aa9b55f39"
server_key_iv: "314dcde0aa96f57cc5581a98f6c103ff"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.429442 4797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:15:56.430006 4811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.430179 4797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:56.430274 4797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "9726c26dd8894fdfb859691ee6cafef3"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "a0fdf79f2d1e397dfd53439aa9b55f39"
server_key_iv: "314dcde0aa96f57cc5581a98f6c103ff"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.430346 4797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:56.439901 4797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:56.441768 4797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:56.441869 4797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:56.442032 4797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:56.442256 4797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:56.442284 4797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.442303 4797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:56.442315 4797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.447904 4797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40901
I20260621 05:15:56.447959 4924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40901 every 8 connection(s)
I20260621 05:15:56.448199 4797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:15:56.450732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 4797
I20260621 05:15:56.450902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:56.451212 392 external_mini_cluster.cc:1428] Setting key 8ad7ddb507341357d77969b0839f7513
Jun 21 05:15:56 dist-test-slave-czm5 krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:56.457381 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.448858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56261 (local address 127.0.98.62:39617)
0621 05:15:56.449252 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.449254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.449266 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:15:56.449581 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.449583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.449595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.449672 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:56.450735 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.451506 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.452645 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.452850 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.452928 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.454943 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:56.454945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:56.454945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:56.454952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:56.456325 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:56.456665 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:56.456666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:56.456666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:56.456714 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:56.456903 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:56.456903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:56.456904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:56.456994 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:56.457051 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.457254 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.457309 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260621 05:15:56.457526 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.448844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39617 (local address 127.0.98.3:56261)
0621 05:15:56.448995 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.449032 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.449488 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.449720 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.449723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.450020 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:56.450622 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.450629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.451577 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.451578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.452560 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.452563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.452938 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.452939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.453062 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.453533 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:56.453540 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:56.454852 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:56.456393 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:56.456399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:56.456400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:56.456600 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:56.456766 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:56.456767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:56.456767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:56.456854 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:56.457051 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:56.457055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:56.457154 (+ 99us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.457229 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.457346 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260621 05:15:56.457934 4925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39617
I20260621 05:15:56.458024 4925 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:56.458160 4925 heartbeater.cc:511] Master 127.0.98.62:39617 requested a full tablet report, sending...
I20260621 05:15:56.458498 4469 ts_manager.cc:194] Registered new tserver with Master: 9726c26dd8894fdfb859691ee6cafef3 (127.0.98.3:40901)
I20260621 05:15:56.458834 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56261
I20260621 05:15:56.462800 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:15:56.469455 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.464001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.0.98.62:39617)
0621 05:15:56.464052 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.464054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.464168 (+ 114us) server_negotiation.cc:407] Connection header received
0621 05:15:56.464239 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.464240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.464249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.464301 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:56.464844 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.465312 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.466032 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.466226 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.466302 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.467319 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:56.467320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:56.467320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:56.467326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:56.468397 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:56.468748 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:56.468750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:56.468750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:56.468800 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:56.468972 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:56.468973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:56.468973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:56.469111 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:56.469202 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.469305 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.469369 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:15:56.470616 4469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:15:56.472203 4469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:56.478127 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.474185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40901 (local address 127.0.0.1:50180)
0621 05:15:56.474342 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.474398 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.474459 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.474809 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.474812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.474814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:56.474874 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.474876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.475931 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.475932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.477060 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.477062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.477810 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.477811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.477897 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.477915 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.477997 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.478058 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":54,"threads_started":1}
I20260621 05:15:56.478143 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.474497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.0.98.3:40901)
0621 05:15:56.474649 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.474653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.474664 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:15:56.474700 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.474702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.474716 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.474794 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:56.474958 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.475835 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.477154 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.477717 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.477807 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.477863 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.478019 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.478066 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260621 05:15:56.478448 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.474189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34389 (local address 127.0.0.1:47346)
0621 05:15:56.474675 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.474683 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.474726 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.474917 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.474918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.474920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:56.474978 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.474979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.476428 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.476429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.477281 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.477283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.478235 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.478236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.478293 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.478297 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.478341 (+ 44us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.478379 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":46,"threads_started":1}
I20260621 05:15:56.478586 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.474185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41075 (local address 127.0.0.1:43648)
0621 05:15:56.474557 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.474566 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.474610 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.475009 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.475010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.475011 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:56.475042 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.475043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.476328 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.476329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.477363 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.477365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.478367 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.478368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.478432 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.478434 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.478475 (+ 41us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.478508 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260621 05:15:56.478732 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.474449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.0.98.1:34389)
0621 05:15:56.474621 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.474625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.474725 (+ 100us) server_negotiation.cc:407] Connection header received
0621 05:15:56.474808 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.474810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.474825 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.474913 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:56.475128 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.476341 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.477358 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.478083 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.478152 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.478200 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.478600 (+ 400us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.478648 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260621 05:15:56.478842 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.474343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.0.98.2:41075)
0621 05:15:56.474473 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.474478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.474801 (+ 323us) server_negotiation.cc:407] Connection header received
0621 05:15:56.474845 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.474848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.474863 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.474932 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:56.475117 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.476228 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.477427 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.478085 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.478151 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.478695 (+ 544us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.478717 (+ 22us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.478755 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260621 05:15:56.478950 4859 tablet_service.cc:1467] Processing CreateTablet for tablet 58523ab7c9c24daca3c73d0ab125ce88 (DEFAULT_TABLE table=test-table [id=58440618135e4b009097839c66ccd45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:56.479308 4859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58523ab7c9c24daca3c73d0ab125ce88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:56.479355 4593 tablet_service.cc:1467] Processing CreateTablet for tablet 58523ab7c9c24daca3c73d0ab125ce88 (DEFAULT_TABLE table=test-table [id=58440618135e4b009097839c66ccd45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:56.479548 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 58523ab7c9c24daca3c73d0ab125ce88 (DEFAULT_TABLE table=test-table [id=58440618135e4b009097839c66ccd45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:56.479691 4593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58523ab7c9c24daca3c73d0ab125ce88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:56.479856 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58523ab7c9c24daca3c73d0ab125ce88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:56.481915 4939 tablet_bootstrap.cc:492] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173: Bootstrap starting.
I20260621 05:15:56.482604 4940 tablet_bootstrap.cc:492] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: Bootstrap starting.
I20260621 05:15:56.482656 4939 tablet_bootstrap.cc:654] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:56.482909 4941 tablet_bootstrap.cc:492] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Bootstrap starting.
I20260621 05:15:56.483188 4940 tablet_bootstrap.cc:654] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:56.483219 4939 log.cc:824] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:56.483515 4941 tablet_bootstrap.cc:654] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:56.483783 4940 log.cc:824] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:56.484113 4939 tablet_bootstrap.cc:492] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173: No bootstrap required, opened a new log
I20260621 05:15:56.484135 4941 log.cc:824] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:56.484217 4939 ts_tablet_manager.cc:1397] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:15:56.484442 4940 tablet_bootstrap.cc:492] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: No bootstrap required, opened a new log
I20260621 05:15:56.484567 4940 ts_tablet_manager.cc:1397] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:15:56.484870 4941 tablet_bootstrap.cc:492] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: No bootstrap required, opened a new log
I20260621 05:15:56.484949 4941 ts_tablet_manager.cc:1397] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:15:56.486195 4939 raft_consensus.cc:348] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.486359 4939 raft_consensus.cc:374] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:56.486295 4940 raft_consensus.cc:348] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.486440 4939 raft_consensus.cc:729] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc0746b2a9c4d23bce9e36d6a823173, State: Initialized, Role: FOLLOWER
I20260621 05:15:56.486479 4940 raft_consensus.cc:374] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:56.486537 4940 raft_consensus.cc:729] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9726c26dd8894fdfb859691ee6cafef3, State: Initialized, Role: FOLLOWER
I20260621 05:15:56.486600 4939 consensus_queue.cc:260] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.486671 4940 consensus_queue.cc:260] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.486863 4939 ts_tablet_manager.cc:1428] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:15:56.486982 4940 ts_tablet_manager.cc:1428] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:15:56.487021 4659 heartbeater.cc:503] Master 127.0.98.62:39617 was elected leader, sending a full tablet report...
I20260621 05:15:56.487020 4925 heartbeater.cc:503] Master 127.0.98.62:39617 was elected leader, sending a full tablet report...
I20260621 05:15:56.487005 4941 raft_consensus.cc:348] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.487138 4941 raft_consensus.cc:374] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:56.487165 4941 raft_consensus.cc:729] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c33d1cdb2846b19816e980a2bb80d3, State: Initialized, Role: FOLLOWER
I20260621 05:15:56.487278 4941 consensus_queue.cc:260] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.487556 4941 ts_tablet_manager.cc:1428] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:15:56.487587 4792 heartbeater.cc:503] Master 127.0.98.62:39617 was elected leader, sending a full tablet report...
W20260621 05:15:56.554986 4793 tablet.cc:2369] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:15:56.567595 4947 raft_consensus.cc:482] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:56.567710 4947 raft_consensus.cc:504] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.568054 4947 leader_election.cc:288] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9726c26dd8894fdfb859691ee6cafef3 (127.0.98.3:40901), fcc0746b2a9c4d23bce9e36d6a823173 (127.0.98.1:34389)
I20260621 05:15:56.570789 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.568245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52673 (local address 127.0.98.3:40901)
0621 05:15:56.568300 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.568302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.568367 (+ 65us) server_negotiation.cc:407] Connection header received
0621 05:15:56.568474 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.568475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.568485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.568561 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:56.568721 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.569161 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.570077 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.570442 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.570528 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.570539 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.570679 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.570715 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:15:56.570801 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.568277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47901 (local address 127.0.98.1:34389)
0621 05:15:56.568317 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.568319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.568525 (+ 206us) server_negotiation.cc:407] Connection header received
0621 05:15:56.568557 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.568558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.568569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.568628 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:15:56.568778 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.569220 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.570105 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.570459 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.570543 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.570552 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.570695 (+ 143us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.570732 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:15:56.570822 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.568196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40901 (local address 127.0.98.2:52673)
0621 05:15:56.568261 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.568275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.568390 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.568567 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.568568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.568570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:56.568639 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.568642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.569249 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.569250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.569963 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.569964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.570526 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.570527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.570605 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.570613 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.570674 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.570715 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 05:15:56.570835 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.568197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34389 (local address 127.0.98.2:47901)
0621 05:15:56.568439 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.568450 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.568510 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.568642 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.568643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.568645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:15:56.568714 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.568716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.569305 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.569306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.570018 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:56.570020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.570544 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.570545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:56.570623 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.570626 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.570688 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.570727 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":53,"thread_start_us":116,"threads_started":1}
I20260621 05:15:56.571148 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58523ab7c9c24daca3c73d0ab125ce88" candidate_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" is_pre_election: true
I20260621 05:15:56.571174 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58523ab7c9c24daca3c73d0ab125ce88" candidate_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9726c26dd8894fdfb859691ee6cafef3" is_pre_election: true
I20260621 05:15:56.571300 4613 raft_consensus.cc:2454] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c33d1cdb2846b19816e980a2bb80d3 in term 0.
I20260621 05:15:56.571316 4879 raft_consensus.cc:2454] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c33d1cdb2846b19816e980a2bb80d3 in term 0.
I20260621 05:15:56.571519 4681 leader_election.cc:302] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c33d1cdb2846b19816e980a2bb80d3, fcc0746b2a9c4d23bce9e36d6a823173; no voters:
I20260621 05:15:56.571643 4947 raft_consensus.cc:2781] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:56.571702 4947 raft_consensus.cc:482] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:56.571724 4947 raft_consensus.cc:3037] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:56.572355 4947 raft_consensus.cc:504] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.572482 4947 leader_election.cc:288] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [CANDIDATE]: Term 1 election: Requested vote from peers 9726c26dd8894fdfb859691ee6cafef3 (127.0.98.3:40901), fcc0746b2a9c4d23bce9e36d6a823173 (127.0.98.1:34389)
I20260621 05:15:56.572614 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58523ab7c9c24daca3c73d0ab125ce88" candidate_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9726c26dd8894fdfb859691ee6cafef3"
I20260621 05:15:56.572667 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58523ab7c9c24daca3c73d0ab125ce88" candidate_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc0746b2a9c4d23bce9e36d6a823173"
I20260621 05:15:56.572691 4879 raft_consensus.cc:3037] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:56.572777 4613 raft_consensus.cc:3037] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:56.573382 4879 raft_consensus.cc:2454] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c33d1cdb2846b19816e980a2bb80d3 in term 1.
I20260621 05:15:56.573408 4613 raft_consensus.cc:2454] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c33d1cdb2846b19816e980a2bb80d3 in term 1.
I20260621 05:15:56.573523 4680 leader_election.cc:302] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9726c26dd8894fdfb859691ee6cafef3, d5c33d1cdb2846b19816e980a2bb80d3; no voters:
I20260621 05:15:56.573626 4947 raft_consensus.cc:2781] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:56.573788 4947 raft_consensus.cc:686] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 1 LEADER]: Becoming Leader. State: Replica: d5c33d1cdb2846b19816e980a2bb80d3, State: Running, Role: LEADER
I20260621 05:15:56.573901 4947 consensus_queue.cc:237] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } }
I20260621 05:15:56.574651 4468 catalog_manager.cc:5495] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c33d1cdb2846b19816e980a2bb80d3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c33d1cdb2846b19816e980a2bb80d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 } health_report { overall_health: UNKNOWN } } }
I20260621 05:15:56.591570 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.589393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.98.2:41075)
0621 05:15:56.589435 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.589437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.589446 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:15:56.589524 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.589525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.589543 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.589614 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:15:56.589728 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.590118 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.590841 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:56.591009 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:56.591083 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:56.591235 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:15:56.591326 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:15:56.591403 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.591442 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.591459 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:15:56.593037 4879 raft_consensus.cc:1261] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 1 FOLLOWER]: Refusing update from remote peer d5c33d1cdb2846b19816e980a2bb80d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:56.593041 4613 raft_consensus.cc:1261] T 58523ab7c9c24daca3c73d0ab125ce88 P fcc0746b2a9c4d23bce9e36d6a823173 [term 1 FOLLOWER]: Refusing update from remote peer d5c33d1cdb2846b19816e980a2bb80d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:56.593376 4947 consensus_queue.cc:1035] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9726c26dd8894fdfb859691ee6cafef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:56.593520 4949 consensus_queue.cc:1035] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcc0746b2a9c4d23bce9e36d6a823173" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:56.595664 4952 mvcc.cc:205] Tried to move back new op lower bound from 7299149646202736640 to 7299149646127210496. Current Snapshot: MvccSnapshot[applied={T|T < 7299149646202736640}]
I20260621 05:15:56.595947 4953 mvcc.cc:205] Tried to move back new op lower bound from 7299149646202736640 to 7299149646127210496. Current Snapshot: MvccSnapshot[applied={T|T < 7299149646202736640}]
I20260621 05:15:56.596216 4954 mvcc.cc:205] Tried to move back new op lower bound from 7299149646202736640 to 7299149646127210496. Current Snapshot: MvccSnapshot[applied={T|T < 7299149646202736640}]
I20260621 05:15:56.598040 4468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:56.598137 4468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:56.598873 4468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c662161941b40d28eddd6a7e57db310: Sending DeleteTablet for 3 replicas of tablet 58523ab7c9c24daca3c73d0ab125ce88
I20260621 05:15:56.599216 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 58523ab7c9c24daca3c73d0ab125ce88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43648
I20260621 05:15:56.599268 4593 tablet_service.cc:1514] Processing DeleteTablet for tablet 58523ab7c9c24daca3c73d0ab125ce88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47346
I20260621 05:15:56.599282 4859 tablet_service.cc:1514] Processing DeleteTablet for tablet 58523ab7c9c24daca3c73d0ab125ce88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50180
I20260621 05:15:56.599556 4962 tablet_replica.cc:330] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: stopping tablet replica
I20260621 05:15:56.599557 4963 tablet_replica.cc:330] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: stopping tablet replica
I20260621 05:15:56.599692 4963 raft_consensus.cc:2229] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:56.599684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 4531
I20260621 05:15:56.600419 4963 raft_consensus.cc:2258] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:56.600549 4962 raft_consensus.cc:2229] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:15:56.600697 4962 raft_consensus.cc:2258] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:56.600888 4963 ts_tablet_manager.cc:1905] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:56.601192 4962 ts_tablet_manager.cc:1905] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:56.602437 4963 ts_tablet_manager.cc:1918] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:15:56.602519 4963 log.cc:1196] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/58523ab7c9c24daca3c73d0ab125ce88
I20260621 05:15:56.602790 4963 ts_tablet_manager.cc:1939] T 58523ab7c9c24daca3c73d0ab125ce88 P 9726c26dd8894fdfb859691ee6cafef3: Deleting consensus metadata
I20260621 05:15:56.602979 4962 ts_tablet_manager.cc:1918] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:15:56.603103 4962 log.cc:1196] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/58523ab7c9c24daca3c73d0ab125ce88
I20260621 05:15:56.603199 4454 catalog_manager.cc:4841] TS 9726c26dd8894fdfb859691ee6cafef3 (127.0.98.3:40901): tablet 58523ab7c9c24daca3c73d0ab125ce88 (table test-table [id=58440618135e4b009097839c66ccd45c]) successfully deleted
I20260621 05:15:56.603395 4962 ts_tablet_manager.cc:1939] T 58523ab7c9c24daca3c73d0ab125ce88 P d5c33d1cdb2846b19816e980a2bb80d3: Deleting consensus metadata
I20260621 05:15:56.603772 4453 catalog_manager.cc:4841] TS d5c33d1cdb2846b19816e980a2bb80d3 (127.0.98.2:41075): tablet 58523ab7c9c24daca3c73d0ab125ce88 (table test-table [id=58440618135e4b009097839c66ccd45c]) successfully deleted
W20260621 05:15:56.606160 4455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34389 (error 108)
I20260621 05:15:56.606379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 4664
I20260621 05:15:56.607277 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.607159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34389 (local address 127.0.0.1:47362)
0621 05:15:56.607211 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.607246 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:15:56.607383 4455 catalog_manager.cc:4584] TS fcc0746b2a9c4d23bce9e36d6a823173 (127.0.98.1:34389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58523ab7c9c24daca3c73d0ab125ce88: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34389: connect: Connection refused (error 111)
I20260621 05:15:56.611150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 4797
I20260621 05:15:56.615777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 4438
2026-06-21T05:15:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1873 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T05:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:56Z Disabled control of system clock
I20260621 05:15:56.641680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34527
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 05:15:56.719458 4969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:56.719676 4969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:56.719698 4969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:56.720758 4969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:56.720790 4969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:56.720803 4969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:56.720814 4969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:56.720826 4969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:56.722023 4969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:56.722187 4969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:56.722385 4969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:56.726806 4977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.726858 4975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:56.726958 4969 server_base.cc:1034] running on GCE node
W20260621 05:15:56.726861 4974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:56.727259 4969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:56.727571 4969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:56.728704 4969 hybrid_clock.cc:648] HybridClock initialized: now 1782018956728687 us; error 34 us; skew 500 ppm
I20260621 05:15:56.729880 4969 webserver.cc:460] Webserver started at http://127.0.98.62:42251/ using document root <none> and password file <none>
I20260621 05:15:56.730094 4969 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:56.730139 4969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:56.730284 4969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:56.731302 4969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "e32c163af80b408ab53921f621663414"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "ec127ec5cbe68b26ecd763b053770e07"
server_key_iv: "048a3d33937aa2e0c8fc35fb875f465c"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.731698 4969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "e32c163af80b408ab53921f621663414"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "ec127ec5cbe68b26ecd763b053770e07"
server_key_iv: "048a3d33937aa2e0c8fc35fb875f465c"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.732878 4969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:56.733475 4983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.733659 4969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:56.733741 4969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "e32c163af80b408ab53921f621663414"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "ec127ec5cbe68b26ecd763b053770e07"
server_key_iv: "048a3d33937aa2e0c8fc35fb875f465c"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.733805 4969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:56.737087 4969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:56.737337 4969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:56.737465 4969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:56.741230 4969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34527
I20260621 05:15:56.741273 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34527 every 8 connection(s)
I20260621 05:15:56.741634 4969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:56.742271 5036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:56.744755 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414: Bootstrap starting.
I20260621 05:15:56.745229 5036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:56.745443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 4969
I20260621 05:15:56.745502 5036 log.cc:824] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:56.745546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:56.745708 392 external_mini_cluster.cc:1428] Setting key c63854efe1cca10cc6fd499a795d242d
I20260621 05:15:56.746254 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414: No bootstrap required, opened a new log
I20260621 05:15:56.747977 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.746162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.98.62:34527)
0621 05:15:56.746349 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.746362 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.746385 (+ 23us) server_negotiation.cc:407] Connection header received
0621 05:15:56.746747 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.746752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.746785 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.746891 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:56.747519 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:56.747522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:56.747523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:56.747527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:56.747560 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:56.747569 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:56.747681 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.747742 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.747801 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260621 05:15:56.748404 5036 raft_consensus.cc:348] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c163af80b408ab53921f621663414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260621 05:15:56.748565 5036 raft_consensus.cc:374] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:56.748636 5036 raft_consensus.cc:729] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32c163af80b408ab53921f621663414, State: Initialized, Role: FOLLOWER
I20260621 05:15:56.748764 5036 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c163af80b408ab53921f621663414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260621 05:15:56.748857 5036 raft_consensus.cc:388] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:56.748895 5036 raft_consensus.cc:482] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:56.748940 5036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:56.749675 5036 raft_consensus.cc:504] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c163af80b408ab53921f621663414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260621 05:15:56.749778 5036 leader_election.cc:302] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32c163af80b408ab53921f621663414; no voters:
I20260621 05:15:56.749923 5036 leader_election.cc:288] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:56.749950 5041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:56.750080 5041 raft_consensus.cc:686] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [term 1 LEADER]: Becoming Leader. State: Replica: e32c163af80b408ab53921f621663414, State: Running, Role: LEADER
I20260621 05:15:56.750169 5036 sys_catalog.cc:564] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:56.750214 5041 consensus_queue.cc:237] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c163af80b408ab53921f621663414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260621 05:15:56.750577 5042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e32c163af80b408ab53921f621663414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c163af80b408ab53921f621663414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } } }
I20260621 05:15:56.750595 5043 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e32c163af80b408ab53921f621663414. Latest consensus state: current_term: 1 leader_uuid: "e32c163af80b408ab53921f621663414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c163af80b408ab53921f621663414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } } }
I20260621 05:15:56.750903 5042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:56.751176 5050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:56.750913 5043 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:56.751662 5050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:56.752991 5050 catalog_manager.cc:1347] Generated new cluster ID: ea7ba043411e4f62b40598796578f158
I20260621 05:15:56.753042 5050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:56.770459 5050 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:56.770628 5050 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:56.776851 5050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414: Generated new TSK 0
I20260621 05:15:56.777011 5050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:15:56.785218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--builtin_ntp_servers=127.0.98.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 05:15:56.862713 5060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:56.862890 5060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:56.862910 5060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:56.864037 5060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:56.864071 5060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:56.864096 5060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:56.865361 5060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:56.865568 5060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:56.865844 5060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:56.870518 5065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.870502 5066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.870499 5068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:56.870668 5060 server_base.cc:1034] running on GCE node
I20260621 05:15:56.870887 5060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:56.871143 5060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:56.872287 5060 hybrid_clock.cc:648] HybridClock initialized: now 1782018956872280 us; error 35 us; skew 500 ppm
I20260621 05:15:56.873431 5060 webserver.cc:460] Webserver started at http://127.0.98.1:43929/ using document root <none> and password file <none>
I20260621 05:15:56.873652 5060 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:56.873693 5060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:56.873828 5060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:56.874804 5060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "ba7d23812ad549b28d6b840484c062e8"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "e75927cfaf98d4f323d251fe5f5668d0"
server_key_iv: "62bb3707ae1009e6126bcb37844c156a"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.875137 5060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ba7d23812ad549b28d6b840484c062e8"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "e75927cfaf98d4f323d251fe5f5668d0"
server_key_iv: "62bb3707ae1009e6126bcb37844c156a"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.876398 5060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:15:56.877048 5074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.877197 5060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:56.877278 5060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "ba7d23812ad549b28d6b840484c062e8"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "e75927cfaf98d4f323d251fe5f5668d0"
server_key_iv: "62bb3707ae1009e6126bcb37844c156a"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.877351 5060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:56.880492 5060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:56.880714 5060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:56.880844 5060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:56.881026 5060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:56.881284 5060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:56.881333 5060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.881371 5060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:56.881398 5060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.886592 5060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38593
I20260621 05:15:56.886629 5187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38593 every 8 connection(s)
I20260621 05:15:56.886922 5060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:15:56.889396 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.887572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44945 (local address 127.0.98.62:34527)
0621 05:15:56.887644 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.887646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.887840 (+ 194us) server_negotiation.cc:407] Connection header received
0621 05:15:56.888209 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.888211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.888217 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.888260 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:56.889198 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:56.889199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:56.889200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:56.889203 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:56.889215 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:56.889218 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:56.889262 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:56.889337 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:15:56.889347 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:15:56.889533 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.887556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34527 (local address 127.0.98.1:44945)
0621 05:15:56.887735 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.887763 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.888138 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.888310 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.888313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.888320 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:56.889077 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:15:56.889077 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:56.889100 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:15:56.889125 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:56.889270 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:56.889273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:56.889275 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:15:56.889323 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:15:56.889366 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260621 05:15:56.889675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 5060
I20260621 05:15:56.889796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:56.889967 392 external_mini_cluster.cc:1428] Setting key cd730de585b2fed909f87bd4757c42fa
I20260621 05:15:56.890107 5188 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34527
I20260621 05:15:56.890198 5188 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:56.890328 5188 heartbeater.cc:511] Master 127.0.98.62:34527 requested a full tablet report, sending...
I20260621 05:15:56.890774 5000 ts_manager.cc:194] Registered new tserver with Master: ba7d23812ad549b28d6b840484c062e8 (127.0.98.1:38593)
I20260621 05:15:56.891256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--builtin_ntp_servers=127.0.98.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 05:15:56.971490 5191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:56.971743 5191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:56.971769 5191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:56.973137 5191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:56.973183 5191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:56.973209 5191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:15:56.974830 5191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:56.975066 5191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:56.975364 5191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:56.980448 5197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.980458 5199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:56.980436 5196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:56.980579 5191 server_base.cc:1034] running on GCE node
I20260621 05:15:56.980818 5191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:56.981061 5191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:56.982203 5191 hybrid_clock.cc:648] HybridClock initialized: now 1782018956982180 us; error 40 us; skew 500 ppm
I20260621 05:15:56.983343 5191 webserver.cc:460] Webserver started at http://127.0.98.2:33857/ using document root <none> and password file <none>
I20260621 05:15:56.983570 5191 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:56.983654 5191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:56.983816 5191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:56.984790 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "73f5b4ce1af643ea9428bdf14dcaf144"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "4135ffb3d03a5bb8174474d1afac2fb2"
server_key_iv: "2b495c4ced96fc48892b54166798de23"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.985139 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "73f5b4ce1af643ea9428bdf14dcaf144"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "4135ffb3d03a5bb8174474d1afac2fb2"
server_key_iv: "2b495c4ced96fc48892b54166798de23"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.986588 5191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:56.987246 5205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.987411 5191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:56.987506 5191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "73f5b4ce1af643ea9428bdf14dcaf144"
format_stamp: "Formatted at 2026-06-21 05:15:56 on dist-test-slave-czm5"
server_key: "4135ffb3d03a5bb8174474d1afac2fb2"
server_key_iv: "2b495c4ced96fc48892b54166798de23"
server_key_version: "encryptionkey@0"
I20260621 05:15:56.987591 5191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:56.991048 5191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:56.991245 5191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:56.991341 5191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:56.991478 5191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:56.991760 5191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:56.991792 5191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.991811 5191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:56.991827 5191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:56.997328 5191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44427
I20260621 05:15:56.997364 5318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44427 every 8 connection(s)
I20260621 05:15:56.997632 5191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:15:57.000193 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.998188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40801 (local address 127.0.98.62:34527)
0621 05:15:56.998233 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:56.998235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:56.998457 (+ 222us) server_negotiation.cc:407] Connection header received
0621 05:15:56.998855 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:56.998856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:56.998860 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:56.998911 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.000005 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.000006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.000006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.000010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.000023 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.000028 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.000082 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.000131 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.000143 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:15:57.000360 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:56.998174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34527 (local address 127.0.98.2:40801)
0621 05:15:56.998351 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:56.998386 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:56.998774 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:56.998954 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:56.998957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:56.998964 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:56.999869 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:15:56.999870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:56.999896 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:15:56.999920 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:57.000081 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:57.000084 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:57.000086 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:15:57.000130 (+ 44us) client_negotiation.cc:240] Negotiation successful
0621 05:15:57.000172 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260621 05:15:57.000841 5319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34527
I20260621 05:15:57.000926 5319 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:57.001029 5319 heartbeater.cc:511] Master 127.0.98.62:34527 requested a full tablet report, sending...
I20260621 05:15:57.001361 5000 ts_manager.cc:194] Registered new tserver with Master: 73f5b4ce1af643ea9428bdf14dcaf144 (127.0.98.2:44427)
I20260621 05:15:57.005399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 5191
I20260621 05:15:57.005474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:15:57.005673 392 external_mini_cluster.cc:1428] Setting key 6b1fd599fa1071923d6e5efb85860598
I20260621 05:15:57.006484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--builtin_ntp_servers=127.0.98.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 05:15:57.084647 5322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:57.084856 5322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:57.084889 5322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:57.085940 5322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:57.085995 5322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:57.086035 5322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:15:57.087383 5322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:57.087648 5322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:57.087946 5322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:57.092586 5328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:57.092592 5330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:57.092674 5322 server_base.cc:1034] running on GCE node
W20260621 05:15:57.092597 5327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:57.092962 5322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:57.093163 5322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:57.094314 5322 hybrid_clock.cc:648] HybridClock initialized: now 1782018957094279 us; error 31 us; skew 500 ppm
I20260621 05:15:57.095445 5322 webserver.cc:460] Webserver started at http://127.0.98.3:36949/ using document root <none> and password file <none>
I20260621 05:15:57.095723 5322 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:57.095765 5322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:57.095897 5322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:57.096886 5322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a012848aff9470aae7a33eb37b56f75"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "a8d8aaf002c8daa5c28188155c5b1d7b"
server_key_iv: "26cf2528daac00dda6d76bc90a24ef7e"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.097226 5322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a012848aff9470aae7a33eb37b56f75"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "a8d8aaf002c8daa5c28188155c5b1d7b"
server_key_iv: "26cf2528daac00dda6d76bc90a24ef7e"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.098385 5322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:57.098971 5336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:57.099109 5322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:57.099180 5322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "4a012848aff9470aae7a33eb37b56f75"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "a8d8aaf002c8daa5c28188155c5b1d7b"
server_key_iv: "26cf2528daac00dda6d76bc90a24ef7e"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.099227 5322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:57.102175 5322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:57.102357 5322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:57.102443 5322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:57.102572 5322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:57.102782 5322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:57.102808 5322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:57.102826 5322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:57.102838 5322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:57.108018 5322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42759
I20260621 05:15:57.108074 5449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42759 every 8 connection(s)
I20260621 05:15:57.108332 5322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:15:57.110786 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.108997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48899 (local address 127.0.98.62:34527)
0621 05:15:57.109056 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.109058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.109268 (+ 210us) server_negotiation.cc:407] Connection header received
0621 05:15:57.109621 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.109622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.109627 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.109677 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.110588 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.110589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.110589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.110593 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.110604 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.110608 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.110651 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.110728 (+ 77us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.110739 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:15:57.110785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 5322
I20260621 05:15:57.110872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:15:57.111012 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.108984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34527 (local address 127.0.98.3:48899)
0621 05:15:57.109138 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.109172 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.109551 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.109719 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.109722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.109730 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:57.110472 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:15:57.110473 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:57.110496 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:15:57.110519 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:57.110653 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:57.110656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:57.110659 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:15:57.110740 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:15:57.110802 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260621 05:15:57.111065 392 external_mini_cluster.cc:1428] Setting key 82f280da28e2f08fe8aba23f76713751
I20260621 05:15:57.111544 5450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34527
I20260621 05:15:57.111665 5450 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:57.111807 5450 heartbeater.cc:511] Master 127.0.98.62:34527 requested a full tablet report, sending...
I20260621 05:15:57.112176 5000 ts_manager.cc:194] Registered new tserver with Master: 4a012848aff9470aae7a33eb37b56f75 (127.0.98.3:42759)
I20260621 05:15:57.122888 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:15:57.125104 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.124121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.0.98.62:34527)
0621 05:15:57.124181 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.124183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.124322 (+ 139us) server_negotiation.cc:407] Connection header received
0621 05:15:57.124357 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.124357 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.124362 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.124406 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.124876 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.124877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.124877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.124880 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.124891 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.124894 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.124939 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.124972 (+ 33us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.124982 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:15:57.126470 5000 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:15:57.127938 5000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:15:57.131282 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.129830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38593 (local address 127.0.0.1:44052)
0621 05:15:57.130120 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.130130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.130175 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.130357 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.130362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.130369 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:57.130870 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:15:57.130871 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:57.130886 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:15:57.130904 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:57.131143 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:57.131148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:57.131151 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:15:57.131208 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:15:57.131222 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":89,"threads_started":1}
I20260621 05:15:57.131302 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.130052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.0.98.1:38593)
0621 05:15:57.130211 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.130216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.130227 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:15:57.130254 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.130256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.130266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.130383 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.130986 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.130992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.130993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.130997 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.131041 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.131086 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.131138 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.131212 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.131228 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260621 05:15:57.131590 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.129806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42759 (local address 127.0.0.1:40696)
0621 05:15:57.129957 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.129993 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.130046 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.130426 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.130426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.130429 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:57.130992 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:15:57.130992 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:57.130999 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:15:57.131011 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:57.131487 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:57.131488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:57.131489 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:15:57.131518 (+ 29us) client_negotiation.cc:240] Negotiation successful
0621 05:15:57.131531 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260621 05:15:57.131785 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.129809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44427 (local address 127.0.0.1:37846)
0621 05:15:57.130108 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.130116 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.130150 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.130743 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.130744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.130746 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:57.131387 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:15:57.131387 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:57.131393 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:15:57.131405 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:57.131665 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:57.131666 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:57.131667 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:15:57.131709 (+ 42us) client_negotiation.cc:240] Negotiation successful
0621 05:15:57.131721 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"spinlock_wait_cycles":384,"thread_start_us":79,"threads_started":1}
I20260621 05:15:57.131956 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.129957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.0.98.3:42759)
0621 05:15:57.130246 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.130249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.130257 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:15:57.130277 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.130279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.130287 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.130904 (+ 617us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.131247 (+ 343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.131249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.131250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.131252 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.131274 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.131282 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.131849 (+ 567us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.131871 (+ 22us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.131886 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":76,"threads_started":1}
I20260621 05:15:57.131973 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.129944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.0.98.2:44427)
0621 05:15:57.130244 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.130248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.130256 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:15:57.130279 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.130281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.130297 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.130386 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.131487 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.131495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.131495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.131500 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.131545 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.131567 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.131639 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.131878 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.131894 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":84,"threads_started":1}
I20260621 05:15:57.132076 5122 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb47adce6d42daa01fd7df8ee8f794 (DEFAULT_TABLE table=test-table [id=f6bec7aacdc64c698ef1e315beff2da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:57.132390 5122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eb47adce6d42daa01fd7df8ee8f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:57.132539 5384 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb47adce6d42daa01fd7df8ee8f794 (DEFAULT_TABLE table=test-table [id=f6bec7aacdc64c698ef1e315beff2da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:57.132853 5384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eb47adce6d42daa01fd7df8ee8f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:57.134770 5464 tablet_bootstrap.cc:492] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8: Bootstrap starting.
I20260621 05:15:57.135375 5253 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb47adce6d42daa01fd7df8ee8f794 (DEFAULT_TABLE table=test-table [id=f6bec7aacdc64c698ef1e315beff2da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:15:57.135561 5464 tablet_bootstrap.cc:654] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:57.135701 5253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eb47adce6d42daa01fd7df8ee8f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:57.135747 5465 tablet_bootstrap.cc:492] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75: Bootstrap starting.
I20260621 05:15:57.135910 5464 log.cc:824] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:57.136358 5465 tablet_bootstrap.cc:654] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:57.136600 5465 log.cc:824] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:57.136652 5464 tablet_bootstrap.cc:492] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8: No bootstrap required, opened a new log
I20260621 05:15:57.136705 5464 ts_tablet_manager.cc:1397] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:57.137185 5465 tablet_bootstrap.cc:492] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75: No bootstrap required, opened a new log
I20260621 05:15:57.137244 5465 ts_tablet_manager.cc:1397] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:57.138947 5468 tablet_bootstrap.cc:492] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: Bootstrap starting.
I20260621 05:15:57.139247 5464 raft_consensus.cc:348] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.139413 5464 raft_consensus.cc:374] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:57.139318 5465 raft_consensus.cc:348] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.139460 5465 raft_consensus.cc:374] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:57.139475 5464 raft_consensus.cc:729] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7d23812ad549b28d6b840484c062e8, State: Initialized, Role: FOLLOWER
I20260621 05:15:57.139580 5468 tablet_bootstrap.cc:654] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:57.139562 5465 raft_consensus.cc:729] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a012848aff9470aae7a33eb37b56f75, State: Initialized, Role: FOLLOWER
I20260621 05:15:57.139596 5464 consensus_queue.cc:260] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.139729 5465 consensus_queue.cc:260] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.139869 5464 ts_tablet_manager.cc:1428] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:15:57.139906 5468 log.cc:824] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:57.139906 5188 heartbeater.cc:503] Master 127.0.98.62:34527 was elected leader, sending a full tablet report...
I20260621 05:15:57.140017 5465 ts_tablet_manager.cc:1428] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:15:57.140056 5450 heartbeater.cc:503] Master 127.0.98.62:34527 was elected leader, sending a full tablet report...
I20260621 05:15:57.140981 5468 tablet_bootstrap.cc:492] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: No bootstrap required, opened a new log
I20260621 05:15:57.141084 5468 ts_tablet_manager.cc:1397] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:15:57.142752 5468 raft_consensus.cc:348] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.142884 5468 raft_consensus.cc:374] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:57.142925 5468 raft_consensus.cc:729] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f5b4ce1af643ea9428bdf14dcaf144, State: Initialized, Role: FOLLOWER
I20260621 05:15:57.143028 5468 consensus_queue.cc:260] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.143252 5468 ts_tablet_manager.cc:1428] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:15:57.143306 5319 heartbeater.cc:503] Master 127.0.98.62:34527 was elected leader, sending a full tablet report...
I20260621 05:15:57.152824 5469 raft_consensus.cc:482] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:15:57.152899 5469 raft_consensus.cc:504] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.153216 5469 leader_election.cc:288] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73f5b4ce1af643ea9428bdf14dcaf144 (127.0.98.2:44427), 4a012848aff9470aae7a33eb37b56f75 (127.0.98.3:42759)
I20260621 05:15:57.154364 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.153374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44427 (local address 127.0.98.1:44023)
0621 05:15:57.153422 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.153434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.153471 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.153639 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.153640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.153642 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:57.154045 (+ 403us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:15:57.154045 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:57.154052 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:15:57.154062 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:57.154210 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:57.154210 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:57.154211 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:15:57.154262 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:15:57.154276 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 05:15:57.154366 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.153473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44023 (local address 127.0.98.2:44427)
0621 05:15:57.153511 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.153513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.153520 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:15:57.153540 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.153541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.153545 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.153603 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.154132 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.154134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.154134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.154138 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.154152 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.154155 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.154211 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.154264 (+ 53us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.154276 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:15:57.154685 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb47adce6d42daa01fd7df8ee8f794" candidate_uuid: "ba7d23812ad549b28d6b840484c062e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" is_pre_election: true
I20260621 05:15:57.154819 5273 raft_consensus.cc:2454] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba7d23812ad549b28d6b840484c062e8 in term 0.
I20260621 05:15:57.154844 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.153381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42759 (local address 127.0.98.1:51173)
0621 05:15:57.153558 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.153566 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.153608 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.153724 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.153724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.153726 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:57.154196 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:15:57.154197 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:57.154203 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:15:57.154216 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:57.154686 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:57.154686 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:57.154687 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:15:57.154748 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:15:57.154761 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":46,"thread_start_us":72,"threads_started":1}
I20260621 05:15:57.154846 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.153500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51173 (local address 127.0.98.3:42759)
0621 05:15:57.153545 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.153546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.153620 (+ 74us) server_negotiation.cc:407] Connection header received
0621 05:15:57.153645 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.153646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.153651 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.153713 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.154328 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.154330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.154348 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.154352 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.154365 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.154369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.154452 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.154758 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.154769 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:15:57.155058 5078 leader_election.cc:302] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73f5b4ce1af643ea9428bdf14dcaf144, ba7d23812ad549b28d6b840484c062e8; no voters:
I20260621 05:15:57.155128 5404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb47adce6d42daa01fd7df8ee8f794" candidate_uuid: "ba7d23812ad549b28d6b840484c062e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a012848aff9470aae7a33eb37b56f75" is_pre_election: true
I20260621 05:15:57.155192 5469 raft_consensus.cc:2781] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:15:57.155251 5469 raft_consensus.cc:482] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:15:57.155287 5469 raft_consensus.cc:3037] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:57.155290 5404 raft_consensus.cc:2454] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba7d23812ad549b28d6b840484c062e8 in term 0.
I20260621 05:15:57.155998 5469 raft_consensus.cc:504] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.156133 5469 leader_election.cc:288] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [CANDIDATE]: Term 1 election: Requested vote from peers 73f5b4ce1af643ea9428bdf14dcaf144 (127.0.98.2:44427), 4a012848aff9470aae7a33eb37b56f75 (127.0.98.3:42759)
I20260621 05:15:57.156322 5404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb47adce6d42daa01fd7df8ee8f794" candidate_uuid: "ba7d23812ad549b28d6b840484c062e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a012848aff9470aae7a33eb37b56f75"
I20260621 05:15:57.156380 5404 raft_consensus.cc:3037] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:57.156344 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb47adce6d42daa01fd7df8ee8f794" candidate_uuid: "ba7d23812ad549b28d6b840484c062e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73f5b4ce1af643ea9428bdf14dcaf144"
I20260621 05:15:57.156426 5273 raft_consensus.cc:3037] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:57.157176 5273 raft_consensus.cc:2454] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba7d23812ad549b28d6b840484c062e8 in term 1.
I20260621 05:15:57.157188 5404 raft_consensus.cc:2454] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba7d23812ad549b28d6b840484c062e8 in term 1.
I20260621 05:15:57.157361 5077 leader_election.cc:302] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a012848aff9470aae7a33eb37b56f75, ba7d23812ad549b28d6b840484c062e8; no voters:
I20260621 05:15:57.157445 5469 raft_consensus.cc:2781] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:57.157580 5469 raft_consensus.cc:686] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [term 1 LEADER]: Becoming Leader. State: Replica: ba7d23812ad549b28d6b840484c062e8, State: Running, Role: LEADER
I20260621 05:15:57.157676 5469 consensus_queue.cc:237] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } }
I20260621 05:15:57.158319 4999 catalog_manager.cc:5495] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba7d23812ad549b28d6b840484c062e8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ba7d23812ad549b28d6b840484c062e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba7d23812ad549b28d6b840484c062e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38593 } health_report { overall_health: HEALTHY } } }
I20260621 05:15:57.167083 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.166303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.0.98.1:38593)
0621 05:15:57.166349 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.166351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.166371 (+ 20us) server_negotiation.cc:407] Connection header received
0621 05:15:57.166420 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.166421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.166430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.166469 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.166927 (+ 458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.166928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.166928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.166931 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.166943 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.166947 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.166991 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.167033 (+ 42us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.167043 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 05:15:57.168597 5404 raft_consensus.cc:1261] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 1 FOLLOWER]: Refusing update from remote peer ba7d23812ad549b28d6b840484c062e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:57.168656 5273 raft_consensus.cc:1261] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 1 FOLLOWER]: Refusing update from remote peer ba7d23812ad549b28d6b840484c062e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:15:57.168913 5474 consensus_queue.cc:1035] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a012848aff9470aae7a33eb37b56f75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:57.169020 5469 consensus_queue.cc:1035] T 79eb47adce6d42daa01fd7df8ee8f794 P ba7d23812ad549b28d6b840484c062e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73f5b4ce1af643ea9428bdf14dcaf144" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:15:57.170981 5477 mvcc.cc:205] Tried to move back new op lower bound from 7299149648560152576 to 7299149648518414336. Current Snapshot: MvccSnapshot[applied={T|T < 7299149648560152576}]
I20260621 05:15:57.171376 5479 mvcc.cc:205] Tried to move back new op lower bound from 7299149648560152576 to 7299149648518414336. Current Snapshot: MvccSnapshot[applied={T|T < 7299149648560152576}]
I20260621 05:15:57.171777 5478 mvcc.cc:205] Tried to move back new op lower bound from 7299149648560152576 to 7299149648518414336. Current Snapshot: MvccSnapshot[applied={T|T < 7299149648560152576}]
I20260621 05:15:57.173072 4999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:57.173141 4999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:15:57.173776 4999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e32c163af80b408ab53921f621663414: Sending DeleteTablet for 3 replicas of tablet 79eb47adce6d42daa01fd7df8ee8f794
I20260621 05:15:57.174100 5122 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eb47adce6d42daa01fd7df8ee8f794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:57 UTC) from {username='slave'} at 127.0.0.1:44052
I20260621 05:15:57.174080 5253 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eb47adce6d42daa01fd7df8ee8f794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:57 UTC) from {username='slave'} at 127.0.0.1:37846
I20260621 05:15:57.174111 5384 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eb47adce6d42daa01fd7df8ee8f794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:15:57 UTC) from {username='slave'} at 127.0.0.1:40696
I20260621 05:15:57.174327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 5060
I20260621 05:15:57.174916 5488 tablet_replica.cc:330] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: stopping tablet replica
I20260621 05:15:57.175036 5488 raft_consensus.cc:2229] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:57.175132 5488 raft_consensus.cc:2258] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:57.175513 5488 ts_tablet_manager.cc:1905] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:15:57.176889 5488 ts_tablet_manager.cc:1918] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:15:57.176966 5488 log.cc:1196] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/79eb47adce6d42daa01fd7df8ee8f794
I20260621 05:15:57.177242 5488 ts_tablet_manager.cc:1939] T 79eb47adce6d42daa01fd7df8ee8f794 P 73f5b4ce1af643ea9428bdf14dcaf144: Deleting consensus metadata
I20260621 05:15:57.177649 4987 catalog_manager.cc:4841] TS 73f5b4ce1af643ea9428bdf14dcaf144 (127.0.98.2:44427): tablet 79eb47adce6d42daa01fd7df8ee8f794 (table test-table [id=f6bec7aacdc64c698ef1e315beff2da3]) successfully deleted
I20260621 05:15:57.179742 5489 tablet_replica.cc:330] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75: stopping tablet replica
I20260621 05:15:57.179842 5489 raft_consensus.cc:2229] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:15:57.179952 5489 raft_consensus.cc:2258] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:15:57.180302 5489 ts_tablet_manager.cc:1905] T 79eb47adce6d42daa01fd7df8ee8f794 P 4a012848aff9470aae7a33eb37b56f75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 05:15:57.180696 4984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38593 (error 108)
I20260621 05:15:57.180845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 5191
I20260621 05:15:57.181582 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.181458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38593 (local address 127.0.0.1:44074)
0621 05:15:57.181511 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.181552 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 05:15:57.181723 4984 catalog_manager.cc:4584] TS ba7d23812ad549b28d6b840484c062e8 (127.0.98.1:38593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79eb47adce6d42daa01fd7df8ee8f794: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38593: connect: Connection refused (error 111)
I20260621 05:15:57.185096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 5322
I20260621 05:15:57.189035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 4969
I20260621 05:15:57.189064 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.188954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42759 (local address 127.0.0.1:40702)
0621 05:15:57.189006 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.189036 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:42759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
2026-06-21T05:15:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (572 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 05:15:57.208029 392 mini_oidc.cc:152] Starting JWKS server
I20260621 05:15:57.209017 392 webserver.cc:460] Webserver started at https://127.0.0.1:46195/ using document root <none> and password file <none>
I20260621 05:15:57.209116 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:15:57.209316 392 webserver.cc:460] Webserver started at http://127.0.0.1:33013/ using document root <none> and password file <none>
2026-06-21T05:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:57Z Disabled control of system clock
I20260621 05:15:57.216684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42187
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42187
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46195/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/testchainca.cert with env {}
W20260621 05:15:57.293016 5496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:57.293185 5496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:57.293205 5496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:57.294215 5496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:57.294246 5496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:57.294258 5496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:57.294269 5496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:57.294281 5496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:15:57.294306 5496 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:15:57.294318 5496 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46195/jwks.json
I20260621 05:15:57.295745 5496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46195/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:57.295909 5496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:57.296072 5496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:57.300418 5502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:57.300413 5501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:57.300413 5504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:57.300541 5496 server_base.cc:1034] running on GCE node
I20260621 05:15:57.300884 5496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:57.301148 5496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:57.302289 5496 hybrid_clock.cc:648] HybridClock initialized: now 1782018957302275 us; error 28 us; skew 500 ppm
I20260621 05:15:57.303382 5496 webserver.cc:460] Webserver started at http://127.0.98.62:39721/ using document root <none> and password file <none>
I20260621 05:15:57.303663 5496 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:57.303722 5496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:57.303864 5496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:57.304802 5496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "3b412704e0b54f3792d56a3fa0a61f5d"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "0787fd17ed234aa4afab6450c78fb6d0"
server_key_iv: "94635190f927279532fa770b926751f7"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.305145 5496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b412704e0b54f3792d56a3fa0a61f5d"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "0787fd17ed234aa4afab6450c78fb6d0"
server_key_iv: "94635190f927279532fa770b926751f7"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.306356 5496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:15:57.307019 5510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:57.307197 5496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:57.307281 5496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "3b412704e0b54f3792d56a3fa0a61f5d"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "0787fd17ed234aa4afab6450c78fb6d0"
server_key_iv: "94635190f927279532fa770b926751f7"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.307346 5496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:57.319500 5496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:57.319782 5496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:57.319917 5496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:57.323305 5496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42187
I20260621 05:15:57.323346 5562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42187 every 8 connection(s)
I20260621 05:15:57.323658 5496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:57.324548 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:57.326521 5563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d: Bootstrap starting.
I20260621 05:15:57.327045 5563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:57.327320 5563 log.cc:824] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:57.328022 5563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d: No bootstrap required, opened a new log
I20260621 05:15:57.329530 5563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b412704e0b54f3792d56a3fa0a61f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42187 } }
I20260621 05:15:57.329674 5563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:57.329720 5563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b412704e0b54f3792d56a3fa0a61f5d, State: Initialized, Role: FOLLOWER
I20260621 05:15:57.329849 5563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b412704e0b54f3792d56a3fa0a61f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42187 } }
I20260621 05:15:57.329926 5563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:57.329965 5563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:57.330031 5563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:57.330312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 5496
I20260621 05:15:57.330399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:57.330588 392 external_mini_cluster.cc:1428] Setting key 2dadd73dc709608e85814e7aeda59cfa
I20260621 05:15:57.330755 5563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b412704e0b54f3792d56a3fa0a61f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42187 } }
I20260621 05:15:57.330873 5563 leader_election.cc:302] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b412704e0b54f3792d56a3fa0a61f5d; no voters:
I20260621 05:15:57.331245 5563 leader_election.cc:288] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:57.331293 5568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:57.331489 5568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [term 1 LEADER]: Becoming Leader. State: Replica: 3b412704e0b54f3792d56a3fa0a61f5d, State: Running, Role: LEADER
I20260621 05:15:57.331537 5563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:57.331674 5568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b412704e0b54f3792d56a3fa0a61f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42187 } }
I20260621 05:15:57.332010 5568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b412704e0b54f3792d56a3fa0a61f5d. Latest consensus state: current_term: 1 leader_uuid: "3b412704e0b54f3792d56a3fa0a61f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b412704e0b54f3792d56a3fa0a61f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42187 } } }
I20260621 05:15:57.332022 5570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b412704e0b54f3792d56a3fa0a61f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b412704e0b54f3792d56a3fa0a61f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42187 } } }
I20260621 05:15:57.332142 5568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:57.332170 5570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:57.332362 5575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:57.332880 5575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:57.334771 5575 catalog_manager.cc:1347] Generated new cluster ID: 3724a7a97f0a42cea05eb0126951d409
I20260621 05:15:57.334826 5575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:57.336726 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.331007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.0.98.62:42187)
0621 05:15:57.331201 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.331207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.331252 (+ 45us) server_negotiation.cc:407] Connection header received
0621 05:15:57.331362 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.331369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.331453 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.331859 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.331948 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.333667 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.334722 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.335306 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.335419 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.336071 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.336074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.336075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.336254 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.336290 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.336302 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.336363 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.336456 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.336524 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260621 05:15:57.337594 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:15:57.363543 5575 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:57.364065 5575 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:57.366592 5575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b412704e0b54f3792d56a3fa0a61f5d: Generated new TSK 0
I20260621 05:15:57.366760 5575 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 05:15:57.391021 5593 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 05:15:57.391105 5593 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 05:15:57.391377 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.376658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.0.98.62:42187)
0621 05:15:57.376719 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.376721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.376853 (+ 132us) server_negotiation.cc:407] Connection header received
0621 05:15:57.376968 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.376969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.376981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.377070 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:15:57.377221 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.377687 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.378502 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.378692 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.378779 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.379010 (+ 231us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:15:57.391136 (+ 12126us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 05:15:57.391307 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260621 05:15:57.391429 5567 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 05:15:57.391705 5592 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:15:57.376601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42187 (local address 127.0.0.1:50108)
0621 05:15:57.376792 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.376800 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.376872 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.377084 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.377085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.377088 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:15:57.377152 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:57.377154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.377822 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.377823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:57.378403 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:57.378405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.378779 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.378782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:57.378870 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.378887 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:15:57.391282 (+ 12395us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 05:15:57.391368 (+ 86us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 05:15:57.392127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 5496
2026-06-21T05:15:57Z chronyd exiting
I20260621 05:15:57.427896 392 test_util.cc:175] -----------------------------------------------
I20260621 05:15:57.427946 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018933478224-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 05:15:57.428574 392 mini_oidc.cc:152] Starting JWKS server
I20260621 05:15:57.429147 392 webserver.cc:460] Webserver started at https://127.0.0.1:41983/ using document root <none> and password file <none>
I20260621 05:15:57.429211 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:15:57.429369 392 webserver.cc:460] Webserver started at http://127.0.0.1:36439/ using document root <none> and password file <none>
2026-06-21T05:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:57Z Disabled control of system clock
I20260621 05:15:57.436111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35447
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35447
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41983/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/testchainca.cert with env {}
W20260621 05:15:57.513355 5599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:57.513564 5599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:57.513598 5599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:57.514760 5599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:57.514825 5599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:57.514846 5599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:57.514866 5599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:57.514897 5599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:15:57.514922 5599 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:15:57.514948 5599 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41983/jwks.json
I20260621 05:15:57.516505 5599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41983/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:57.516770 5599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:57.516970 5599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:57.521983 5604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:57.521993 5605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:57.522002 5599 server_base.cc:1034] running on GCE node
W20260621 05:15:57.521988 5607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:57.522441 5599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:57.522754 5599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:57.523902 5599 hybrid_clock.cc:648] HybridClock initialized: now 1782018957523882 us; error 33 us; skew 500 ppm
I20260621 05:15:57.524914 5599 webserver.cc:460] Webserver started at http://127.0.98.62:39497/ using document root <none> and password file <none>
I20260621 05:15:57.525161 5599 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:57.525228 5599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:57.525352 5599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:57.526307 5599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "560eac559aac41cda14922ebbc71785d"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "fcd1c5dc8bb55ee2fa2340498852ec9e"
server_key_iv: "2218004d5198b02e70c582931ed370e0"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.526672 5599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "560eac559aac41cda14922ebbc71785d"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "fcd1c5dc8bb55ee2fa2340498852ec9e"
server_key_iv: "2218004d5198b02e70c582931ed370e0"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.527993 5599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:15:57.528628 5613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:57.528827 5599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:57.528923 5599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "560eac559aac41cda14922ebbc71785d"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "fcd1c5dc8bb55ee2fa2340498852ec9e"
server_key_iv: "2218004d5198b02e70c582931ed370e0"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.529013 5599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:57.540397 5599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:57.540638 5599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:57.540759 5599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:57.544306 5599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35447
I20260621 05:15:57.544332 5665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35447 every 8 connection(s)
I20260621 05:15:57.544693 5599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:57.545603 5666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:57.547729 5666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d: Bootstrap starting.
I20260621 05:15:57.548235 5666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:57.548517 5666 log.cc:824] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:57.549144 5666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d: No bootstrap required, opened a new log
I20260621 05:15:57.550060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 5599
I20260621 05:15:57.550133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:57.550278 392 external_mini_cluster.cc:1428] Setting key d6fbeff6a19f74c8d0096a63a278c6b4
I20260621 05:15:57.550993 5666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560eac559aac41cda14922ebbc71785d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } }
I20260621 05:15:57.551169 5666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:57.551203 5666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560eac559aac41cda14922ebbc71785d, State: Initialized, Role: FOLLOWER
I20260621 05:15:57.551321 5666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560eac559aac41cda14922ebbc71785d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } }
I20260621 05:15:57.551401 5666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:57.551430 5666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:57.551474 5666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:57.552145 5666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560eac559aac41cda14922ebbc71785d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } }
I20260621 05:15:57.552241 5666 leader_election.cc:302] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560eac559aac41cda14922ebbc71785d; no voters:
I20260621 05:15:57.552404 5666 leader_election.cc:288] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:57.552453 5671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:57.552605 5666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:57.552618 5671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [term 1 LEADER]: Becoming Leader. State: Replica: 560eac559aac41cda14922ebbc71785d, State: Running, Role: LEADER
I20260621 05:15:57.552843 5671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560eac559aac41cda14922ebbc71785d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } }
I20260621 05:15:57.553196 5673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 560eac559aac41cda14922ebbc71785d. Latest consensus state: current_term: 1 leader_uuid: "560eac559aac41cda14922ebbc71785d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560eac559aac41cda14922ebbc71785d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } } }
I20260621 05:15:57.553270 5673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:57.553411 5672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "560eac559aac41cda14922ebbc71785d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560eac559aac41cda14922ebbc71785d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } } }
I20260621 05:15:57.553472 5672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:57.553823 5680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:57.554306 5680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:57.555913 5680 catalog_manager.cc:1347] Generated new cluster ID: e053a4b3eb6442f086334892652bef5b
I20260621 05:15:57.555992 5680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:57.556257 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.550757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.0.98.62:35447)
0621 05:15:57.550961 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.550968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.550997 (+ 29us) server_negotiation.cc:407] Connection header received
0621 05:15:57.551111 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.551117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.551182 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.551314 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.551560 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.553205 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.554435 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.554835 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.554962 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.555577 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.555581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.555581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.555782 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.555819 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.555832 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.555900 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.555973 (+ 73us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.556040 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260621 05:15:57.557173 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:15:57.569046 5680 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:57.569672 5680 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:57.578020 5680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 560eac559aac41cda14922ebbc71785d: Generated new TSK 0
I20260621 05:15:57.578189 5680 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 05:15:57.605770 5696 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 05:15:57.605834 5696 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 05:15:57.606078 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.592549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.0.98.62:35447)
0621 05:15:57.592598 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.592600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.592691 (+ 91us) server_negotiation.cc:407] Connection header received
0621 05:15:57.592735 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.592735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.592747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.592817 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:15:57.592912 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.593648 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.594380 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.594700 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.594767 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.594891 (+ 124us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:15:57.605899 (+ 11008us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 05:15:57.606024 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":30}
W20260621 05:15:57.606122 5670 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260621 05:15:57.606170 5695 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:15:57.592477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35447 (local address 127.0.0.1:52996)
0621 05:15:57.592625 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.592635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.592689 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.592820 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.592821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.592823 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:15:57.592866 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:57.592867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.593716 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.593717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:57.594280 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:57.594281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.594771 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.594774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:57.594811 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.594815 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:15:57.605996 (+ 11181us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 05:15:57.606079 (+ 83us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35447: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260621 05:15:57.606514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 5599
2026-06-21T05:15:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 05:15:57.643487 392 mini_oidc.cc:152] Starting JWKS server
I20260621 05:15:57.644122 392 webserver.cc:460] Webserver started at https://127.0.0.1:33625/ using document root <none> and password file <none>
I20260621 05:15:57.644160 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:15:57.644337 392 webserver.cc:460] Webserver started at http://127.0.0.1:35501/ using document root <none> and password file <none>
2026-06-21T05:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:57Z Disabled control of system clock
I20260621 05:15:57.651279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39485
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39485
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33625/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 05:15:57.726573 5702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:57.726794 5702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:57.726842 5702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:57.728088 5702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:57.728129 5702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:57.728142 5702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:57.728155 5702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:57.728168 5702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:15:57.728179 5702 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:15:57.728196 5702 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33625/jwks.json
I20260621 05:15:57.729439 5702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33625/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:57.729595 5702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:57.729799 5702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:57.734079 5710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:57.734089 5708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:57.734093 5707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:57.734478 5702 server_base.cc:1034] running on GCE node
I20260621 05:15:57.734709 5702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:57.735004 5702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:57.736156 5702 hybrid_clock.cc:648] HybridClock initialized: now 1782018957736136 us; error 31 us; skew 500 ppm
I20260621 05:15:57.737249 5702 webserver.cc:460] Webserver started at http://127.0.98.62:46261/ using document root <none> and password file <none>
I20260621 05:15:57.737505 5702 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:57.737571 5702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:57.737710 5702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:57.738646 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "0648806e35924d1ea2797a846fdd9ed0"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "445e8ffc234c97314a574e9ed942df77"
server_key_iv: "76432fe4208c37566afedc0641158e5d"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.738977 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "0648806e35924d1ea2797a846fdd9ed0"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "445e8ffc234c97314a574e9ed942df77"
server_key_iv: "76432fe4208c37566afedc0641158e5d"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.740274 5702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:15:57.740885 5716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:57.741026 5702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:15:57.741115 5702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "0648806e35924d1ea2797a846fdd9ed0"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "445e8ffc234c97314a574e9ed942df77"
server_key_iv: "76432fe4208c37566afedc0641158e5d"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.741185 5702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:57.757010 5702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:57.757263 5702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:57.757407 5702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:57.760982 5702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39485
I20260621 05:15:57.761029 5768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39485 every 8 connection(s)
I20260621 05:15:57.761365 5702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:57.761950 5769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:57.764240 5769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0: Bootstrap starting.
I20260621 05:15:57.764793 5769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:57.764894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 5702
I20260621 05:15:57.764986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:57.765116 5769 log.cc:824] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:57.765143 392 external_mini_cluster.cc:1428] Setting key 6e74a5d60966bd1b607d64b4f368f55d
I20260621 05:15:57.765767 5769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0: No bootstrap required, opened a new log
I20260621 05:15:57.767997 5769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0648806e35924d1ea2797a846fdd9ed0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39485 } }
I20260621 05:15:57.768144 5769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:57.768174 5769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0648806e35924d1ea2797a846fdd9ed0, State: Initialized, Role: FOLLOWER
I20260621 05:15:57.768280 5769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0648806e35924d1ea2797a846fdd9ed0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39485 } }
I20260621 05:15:57.768344 5769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:57.768373 5769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:57.768414 5769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:57.768930 5769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0648806e35924d1ea2797a846fdd9ed0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39485 } }
I20260621 05:15:57.769030 5769 leader_election.cc:302] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0648806e35924d1ea2797a846fdd9ed0; no voters:
I20260621 05:15:57.769179 5769 leader_election.cc:288] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:57.769338 5775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:57.769443 5775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [term 1 LEADER]: Becoming Leader. State: Replica: 0648806e35924d1ea2797a846fdd9ed0, State: Running, Role: LEADER
I20260621 05:15:57.769445 5769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:57.770232 5773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.765539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.0.98.62:39485)
0621 05:15:57.765713 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.765724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.765744 (+ 20us) server_negotiation.cc:407] Connection header received
0621 05:15:57.766125 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.766131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.766179 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.766296 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.766506 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.767312 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.768471 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.768860 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.768949 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.769601 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.769605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.769605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.769768 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.769822 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.769832 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.769860 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.770013 (+ 153us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.770083 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260621 05:15:57.770817 5775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0648806e35924d1ea2797a846fdd9ed0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39485 } }
I20260621 05:15:57.771059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:15:57.771286 5776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0648806e35924d1ea2797a846fdd9ed0. Latest consensus state: current_term: 1 leader_uuid: "0648806e35924d1ea2797a846fdd9ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0648806e35924d1ea2797a846fdd9ed0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39485 } } }
I20260621 05:15:57.771459 5774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0648806e35924d1ea2797a846fdd9ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0648806e35924d1ea2797a846fdd9ed0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39485 } } }
W20260621 05:15:57.771664 5788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260621 05:15:57.771809 5776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:57.771821 5774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0 [sys.catalog]: This master's current role is: LEADER
W20260621 05:15:57.771832 5788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:15:57.772207 5791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:57.772359 5791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:57.773563 5791 catalog_manager.cc:1347] Generated new cluster ID: a12fef876ff541318cd960ea4454d5a1
I20260621 05:15:57.773612 5791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:57.797394 5791 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:57.798070 5791 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:57.808305 5791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0648806e35924d1ea2797a846fdd9ed0: Generated new TSK 0
I20260621 05:15:57.808490 5791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:15:57.824640 5773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.821367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.0.98.62:39485)
0621 05:15:57.821420 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.821421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.821516 (+ 95us) server_negotiation.cc:407] Connection header received
0621 05:15:57.821582 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.821582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.821592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.821643 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:15:57.821771 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.822173 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.823215 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.823376 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.823420 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.823598 (+ 178us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:15:57.824461 (+ 863us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33625/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 05:15:57.824579 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33396: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33625/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260621 05:15:57.824683 5773 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33396: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33625/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260621 05:15:57.824762 5798 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:15:57.821315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39485 (local address 127.0.0.1:33396)
0621 05:15:57.821452 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:57.821461 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:57.821522 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:57.821658 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:57.821659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:57.821660 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:15:57.821712 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:57.821714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.822244 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.822245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:57.823143 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:57.823145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.823440 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.823443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:57.823500 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.823505 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:15:57.824559 (+ 1054us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33625/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 05:15:57.824652 (+ 93us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33625/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260621 05:15:57.825173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 5702
E20260621 05:15:57.834751 5799 webserver.cc:533] Webserver: sslize failed:
E20260621 05:15:57.834800 5799 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-21T05:15:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T05:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:57Z Disabled control of system clock
I20260621 05:15:57.869426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37965
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37965
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/testchainca.cert with env {}
W20260621 05:15:57.943984 5803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:57.944168 5803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:57.944200 5803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:57.945317 5803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:57.945377 5803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:57.945397 5803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:57.945434 5803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:57.945446 5803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:15:57.945457 5803 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:15:57.945508 5803 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 05:15:57.946916 5803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:57.947091 5803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:57.947235 5803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:57.951756 5809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:57.951781 5811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:57.951810 5803 server_base.cc:1034] running on GCE node
W20260621 05:15:57.951781 5808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:57.952283 5803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:57.952508 5803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:57.953655 5803 hybrid_clock.cc:648] HybridClock initialized: now 1782018957953628 us; error 43 us; skew 500 ppm
I20260621 05:15:57.954757 5803 webserver.cc:460] Webserver started at http://127.0.98.62:41043/ using document root <none> and password file <none>
I20260621 05:15:57.954983 5803 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:57.955027 5803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:57.955188 5803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:57.956209 5803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "2e4ec195039d442c9a0acfe2f97f2b3f"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "dc7202baf71afff33c79aae396d0a9f7"
server_key_iv: "a6b420da28d69fc0ce38a4fa4966f31e"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.956622 5803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e4ec195039d442c9a0acfe2f97f2b3f"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "dc7202baf71afff33c79aae396d0a9f7"
server_key_iv: "a6b420da28d69fc0ce38a4fa4966f31e"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.957815 5803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:15:57.958460 5817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:57.958632 5803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:57.958743 5803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "2e4ec195039d442c9a0acfe2f97f2b3f"
format_stamp: "Formatted at 2026-06-21 05:15:57 on dist-test-slave-czm5"
server_key: "dc7202baf71afff33c79aae396d0a9f7"
server_key_iv: "a6b420da28d69fc0ce38a4fa4966f31e"
server_key_version: "encryptionkey@0"
I20260621 05:15:57.958822 5803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:57.972379 5803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:57.972627 5803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:57.972766 5803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:57.976372 5803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37965
I20260621 05:15:57.976401 5869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37965 every 8 connection(s)
I20260621 05:15:57.976701 5803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:57.977563 5870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:57.979465 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f: Bootstrap starting.
I20260621 05:15:57.979993 5870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:57.980283 5870 log.cc:824] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:57.980865 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f: No bootstrap required, opened a new log
I20260621 05:15:57.982298 5870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4ec195039d442c9a0acfe2f97f2b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37965 } }
I20260621 05:15:57.982416 5870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:57.982439 5870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4ec195039d442c9a0acfe2f97f2b3f, State: Initialized, Role: FOLLOWER
I20260621 05:15:57.982628 5870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4ec195039d442c9a0acfe2f97f2b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37965 } }
I20260621 05:15:57.982702 5870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:57.982743 5870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:57.982786 5870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:57.983132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 5803
I20260621 05:15:57.983232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:57.983414 392 external_mini_cluster.cc:1428] Setting key f6582890dd30d5d9165380c9bcfa83dd
I20260621 05:15:57.983505 5870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4ec195039d442c9a0acfe2f97f2b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37965 } }
I20260621 05:15:57.983611 5870 leader_election.cc:302] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e4ec195039d442c9a0acfe2f97f2b3f; no voters:
I20260621 05:15:57.983764 5870 leader_election.cc:288] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:57.983784 5873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:57.983964 5873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [term 1 LEADER]: Becoming Leader. State: Replica: 2e4ec195039d442c9a0acfe2f97f2b3f, State: Running, Role: LEADER
I20260621 05:15:57.984056 5870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:57.984071 5873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4ec195039d442c9a0acfe2f97f2b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37965 } }
I20260621 05:15:57.984371 5873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4ec195039d442c9a0acfe2f97f2b3f. Latest consensus state: current_term: 1 leader_uuid: "2e4ec195039d442c9a0acfe2f97f2b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4ec195039d442c9a0acfe2f97f2b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37965 } } }
I20260621 05:15:57.984397 5877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e4ec195039d442c9a0acfe2f97f2b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4ec195039d442c9a0acfe2f97f2b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37965 } } }
I20260621 05:15:57.984534 5877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:57.984520 5873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:57.985122 5884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:57.985577 5884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:15:57.987005 5884 catalog_manager.cc:1347] Generated new cluster ID: db80aa4170de4d019379a540180e4dba
I20260621 05:15:57.987063 5884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:57.989113 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:57.983837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.0.98.62:37965)
0621 05:15:57.984117 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:57.984125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:57.984153 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:15:57.984248 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:57.984253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:57.984306 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:57.984651 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:57.984730 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.986072 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.987134 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:57.987790 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:57.987896 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:57.988501 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:57.988506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:57.988506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:15:57.988699 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:57.988734 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:15:57.988746 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:57.988804 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:57.988886 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 05:15:57.988946 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":70,"threads_started":1}
I20260621 05:15:57.989987 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:15:57.992274 5884 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:57.992689 5884 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:58.001405 5884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e4ec195039d442c9a0acfe2f97f2b3f: Generated new TSK 0
I20260621 05:15:58.001549 5884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:15:58.074550 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:58.009286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.0.98.62:37965)
0621 05:15:58.009337 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:58.009339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:58.009436 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:15:58.009499 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:58.009499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:58.009508 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:58.009561 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:15:58.009672 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:58.010124 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:58.011210 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:58.011388 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:58.011432 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:58.011619 (+ 187us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:15:58.074327 (+ 62708us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 05:15:58.074492 (+ 165us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36822: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260621 05:15:58.074632 5875 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36822: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 05:15:58.074777 5899 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:15:58.009231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37965 (local address 127.0.0.1:36822)
0621 05:15:58.009369 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:58.009379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:58.009439 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:58.009570 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:58.009571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:58.009572 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:15:58.009615 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:58.009616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:58.010214 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:58.010215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:58.011134 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:58.011136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:58.011459 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:58.011462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:58.011523 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:58.011528 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:15:58.074511 (+ 62983us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 05:15:58.074671 (+ 160us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260621 05:15:58.075078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 5803
2026-06-21T05:15:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (229 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (82 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:15:58 dist-test-slave-czm5 krb5kdc[5911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:15:58 dist-test-slave-czm5 krb5kdc[5911](info): set up 2 sockets
Jun 21 05:15:58 dist-test-slave-czm5 krb5kdc[5911](info): commencing operation
krb5kdc: starting...
W20260621 05:15:59.227957 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:15:59 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:15:59.332641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:59.406850 5924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:59.407055 5924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:59.407088 5924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:59.408209 5924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:15:59.408270 5924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:59.408293 5924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:59.408317 5924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:15:59.408344 5924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:15:59.409766 5924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:59.410006 5924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:59.410197 5924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:59.414388 5929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:59.414381 5932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:59.414347 5930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:59.414412 5924 server_base.cc:1034] running on GCE node
I20260621 05:15:59.414799 5924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:59.415071 5924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:59.416181 5924 hybrid_clock.cc:648] HybridClock initialized: now 1782018959416176 us; error 35 us; skew 500 ppm
Jun 21 05:15:59 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:59.418473 5924 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:15:59.418846 5924 webserver.cc:460] Webserver started at http://127.0.98.62:38639/ using document root <none> and password file <none>
I20260621 05:15:59.419059 5924 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:59.419119 5924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:59.419267 5924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:59.420245 5924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "249a405ff7f44e7884b630a23955bea5"
format_stamp: "Formatted at 2026-06-21 05:15:59 on dist-test-slave-czm5"
server_key: "c827869f0c7f70ddbbe2fe5cca4365a6"
server_key_iv: "6e0bf8db87d923d1db399ba75b7632b8"
server_key_version: "encryptionkey@0"
I20260621 05:15:59.420573 5924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "249a405ff7f44e7884b630a23955bea5"
format_stamp: "Formatted at 2026-06-21 05:15:59 on dist-test-slave-czm5"
server_key: "c827869f0c7f70ddbbe2fe5cca4365a6"
server_key_iv: "6e0bf8db87d923d1db399ba75b7632b8"
server_key_version: "encryptionkey@0"
I20260621 05:15:59.421705 5924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:15:59.422294 5939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:59.422458 5924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:59.422540 5924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "249a405ff7f44e7884b630a23955bea5"
format_stamp: "Formatted at 2026-06-21 05:15:59 on dist-test-slave-czm5"
server_key: "c827869f0c7f70ddbbe2fe5cca4365a6"
server_key_iv: "6e0bf8db87d923d1db399ba75b7632b8"
server_key_version: "encryptionkey@0"
I20260621 05:15:59.422636 5924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:59.448282 5924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:59.450245 5924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:59.450392 5924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:59.453784 5924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44217
I20260621 05:15:59.453814 5991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44217 every 8 connection(s)
I20260621 05:15:59.454267 5924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:15:59.454869 5992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:15:59.456821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 5924
I20260621 05:15:59.456933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:15:59.457118 392 external_mini_cluster.cc:1428] Setting key e20dacb526555af791c8d476e0694f8c
I20260621 05:15:59.457338 5992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5: Bootstrap starting.
I20260621 05:15:59.457960 5992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5: Neither blocks nor log segments found. Creating new log.
I20260621 05:15:59.458217 5992 log.cc:824] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:15:59.458848 5992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5: No bootstrap required, opened a new log
I20260621 05:15:59.461054 5992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249a405ff7f44e7884b630a23955bea5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260621 05:15:59.461201 5992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:15:59.461228 5992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249a405ff7f44e7884b630a23955bea5, State: Initialized, Role: FOLLOWER
I20260621 05:15:59.461387 5992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249a405ff7f44e7884b630a23955bea5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260621 05:15:59.461532 5992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:15:59.461581 5992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:15:59.461617 5992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:15:59.462239 5992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249a405ff7f44e7884b630a23955bea5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260621 05:15:59.462342 5992 leader_election.cc:302] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 249a405ff7f44e7884b630a23955bea5; no voters:
I20260621 05:15:59.462469 5992 leader_election.cc:288] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:15:59.462505 5997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:15:59.462611 5997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [term 1 LEADER]: Becoming Leader. State: Replica: 249a405ff7f44e7884b630a23955bea5, State: Running, Role: LEADER
I20260621 05:15:59.462693 5992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:15:59.462724 5997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249a405ff7f44e7884b630a23955bea5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260621 05:15:59.463049 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "249a405ff7f44e7884b630a23955bea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249a405ff7f44e7884b630a23955bea5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } } }
I20260621 05:15:59.463179 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:59.463371 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249a405ff7f44e7884b630a23955bea5. Latest consensus state: current_term: 1 leader_uuid: "249a405ff7f44e7884b630a23955bea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249a405ff7f44e7884b630a23955bea5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } } }
I20260621 05:15:59.463491 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:15:59.464443 6005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:15:59.465054 6005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:15:59 dist-test-slave-czm5 krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:59.466827 6005 catalog_manager.cc:1347] Generated new cluster ID: bf40ffc4362b4404a2598a69ea475d54
I20260621 05:15:59.466885 6005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:15:59.469779 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:59.457550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.0.98.62:44217)
0621 05:15:59.457730 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:59.457739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:59.457770 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:15:59.458218 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:59.458224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:59.458263 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:59.458400 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:59.459144 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:59.460404 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:59.461441 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:59.462023 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:59.462132 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:59.465736 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:59.465740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:59.465741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:59.465751 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:59.467918 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:59.468310 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:59.468313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:59.468313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:59.468434 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:59.468614 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:59.468615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:59.468615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:59.468898 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:59.469035 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:59.469507 (+ 472us) server_negotiation.cc:299] Negotiation successful
0621 05:15:59.469636 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260621 05:15:59.474447 6005 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:15:59.475090 6005 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:15:59.480867 6005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 249a405ff7f44e7884b630a23955bea5: Generated new TSK 0
I20260621 05:15:59.481020 6005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260621 05:15:59.857486 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.373s user 0.001s sys 0.002s
I20260621 05:15:59.857847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:15:59.933995 6018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:15:59.934175 6018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:15:59.934195 6018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:15:59.935288 6018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:15:59.935323 6018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:15:59.935348 6018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:15:59.936785 6018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:15:59.936949 6018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:15:59.937098 6018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:15:59.941788 6023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:59.941939 6018 server_base.cc:1034] running on GCE node
W20260621 05:15:59.941788 6026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:15:59.941795 6024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:15:59.942265 6018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:15:59.942515 6018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:15:59.943689 6018 hybrid_clock.cc:648] HybridClock initialized: now 1782018959943692 us; error 44 us; skew 500 ppm
Jun 21 05:15:59 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:15:59.945940 6018 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:15:59.946388 6018 webserver.cc:460] Webserver started at http://127.0.98.1:33543/ using document root <none> and password file <none>
I20260621 05:15:59.946599 6018 fs_manager.cc:359] Metadata directory not provided
I20260621 05:15:59.946664 6018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:15:59.946796 6018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:15:59.947809 6018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "58d2de2ef3714f6bbfd8dff683a69a22"
format_stamp: "Formatted at 2026-06-21 05:15:59 on dist-test-slave-czm5"
server_key: "1a74d18c9bfbf1d7ebe7a0e325b508e3"
server_key_iv: "481bb9c2dbb2e96f0cf135df40e8e470"
server_key_version: "encryptionkey@0"
I20260621 05:15:59.948148 6018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "58d2de2ef3714f6bbfd8dff683a69a22"
format_stamp: "Formatted at 2026-06-21 05:15:59 on dist-test-slave-czm5"
server_key: "1a74d18c9bfbf1d7ebe7a0e325b508e3"
server_key_iv: "481bb9c2dbb2e96f0cf135df40e8e470"
server_key_version: "encryptionkey@0"
I20260621 05:15:59.949342 6018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:15:59.950075 6033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:59.950240 6018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:15:59.950330 6018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "58d2de2ef3714f6bbfd8dff683a69a22"
format_stamp: "Formatted at 2026-06-21 05:15:59 on dist-test-slave-czm5"
server_key: "1a74d18c9bfbf1d7ebe7a0e325b508e3"
server_key_iv: "481bb9c2dbb2e96f0cf135df40e8e470"
server_key_version: "encryptionkey@0"
I20260621 05:15:59.950408 6018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:15:59.968983 6018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:15:59.970825 6018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:15:59.970971 6018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:15:59.971179 6018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:15:59.971451 6018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:15:59.971503 6018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:59.971545 6018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:15:59.971576 6018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:15:59.977077 6018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34041
I20260621 05:15:59.977111 6146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34041 every 8 connection(s)
I20260621 05:15:59.977387 6018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:15:59.982427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 6018
I20260621 05:15:59.982519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:15:59.982709 392 external_mini_cluster.cc:1428] Setting key 305efba6b1d1dbfdc1cd8ac90f9f22c9
Jun 21 05:15:59 dist-test-slave-czm5 krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:15:59.986485 6150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:59.978017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:38169)
0621 05:15:59.978183 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 05:15:59.978219 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:15:59.979043 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:15:59.979241 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:15:59.979244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:15:59.979457 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:15:59.979682 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:59.979687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:59.980305 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:59.980306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:59.981237 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:15:59.981240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:15:59.981544 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:15:59.981544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:15:59.981635 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:59.982092 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:15:59.982099 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:15:59.983508 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:15:59.985157 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:59.985163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:59.985165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:59.985469 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:59.985706 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:15:59.985707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:15:59.985707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:15:59.985809 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:15:59.986074 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:15:59.986078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:15:59.986174 (+ 96us) client_negotiation.cc:769] Sending connection context
0621 05:15:59.986248 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:15:59.986349 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260621 05:15:59.986544 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:15:59.978034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38169 (local address 127.0.98.62:44217)
0621 05:15:59.978132 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0621 05:15:59.978134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:15:59.978302 (+ 168us) server_negotiation.cc:407] Connection header received
0621 05:15:59.979124 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:15:59.979125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:15:59.979134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:15:59.979187 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:15:59.979775 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:59.980235 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:59.981294 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:15:59.981478 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:15:59.981528 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:15:59.983587 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:15:59.983588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:15:59.983589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:15:59.983597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:15:59.985065 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:59.985549 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:59.985550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:59.985550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:59.985624 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:15:59.985871 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:15:59.985873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:15:59.985873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:15:59.986008 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:15:59.986077 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:15:59.986292 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 05:15:59.986377 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":60,"threads_started":1}
I20260621 05:15:59.986946 6147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260621 05:15:59.987032 6147 heartbeater.cc:464] Registering TS with master...
I20260621 05:15:59.987195 6147 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
I20260621 05:15:59.987731 5956 ts_manager.cc:194] Registered new tserver with Master: 58d2de2ef3714f6bbfd8dff683a69a22 (127.0.98.1:34041)
I20260621 05:15:59.988654 5956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38169
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:16:00.004446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:00.080873 6153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:00.081050 6153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:00.081072 6153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:00.082295 6153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:00.082345 6153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:00.082369 6153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:00.083753 6153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:00.083946 6153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:00.084134 6153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:00.088696 6161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:00.088718 6158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:00.088765 6153 server_base.cc:1034] running on GCE node
W20260621 05:16:00.088703 6159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:00.089174 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:00.089407 6153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:00.090566 6153 hybrid_clock.cc:648] HybridClock initialized: now 1782018960090552 us; error 45 us; skew 500 ppm
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.092927 6153 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:16:00.093273 6153 webserver.cc:460] Webserver started at http://127.0.98.2:34035/ using document root <none> and password file <none>
I20260621 05:16:00.093434 6153 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:00.093470 6153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:00.093554 6153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:00.094444 6153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "a02e17fd5c414ba1b4d2ac2d199db629"
format_stamp: "Formatted at 2026-06-21 05:16:00 on dist-test-slave-czm5"
server_key: "5b3e7c82729a52a4fd1e7e3331e295ba"
server_key_iv: "2a17518168e028c2e325fde82e4cc441"
server_key_version: "encryptionkey@0"
I20260621 05:16:00.094731 6153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a02e17fd5c414ba1b4d2ac2d199db629"
format_stamp: "Formatted at 2026-06-21 05:16:00 on dist-test-slave-czm5"
server_key: "5b3e7c82729a52a4fd1e7e3331e295ba"
server_key_iv: "2a17518168e028c2e325fde82e4cc441"
server_key_version: "encryptionkey@0"
I20260621 05:16:00.095996 6153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:16:00.096638 6168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:00.096787 6153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:00.096868 6153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "a02e17fd5c414ba1b4d2ac2d199db629"
format_stamp: "Formatted at 2026-06-21 05:16:00 on dist-test-slave-czm5"
server_key: "5b3e7c82729a52a4fd1e7e3331e295ba"
server_key_iv: "2a17518168e028c2e325fde82e4cc441"
server_key_version: "encryptionkey@0"
I20260621 05:16:00.096917 6153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:00.108928 6153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:00.110491 6153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:00.110581 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:00.110721 6153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:00.110973 6153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:00.111001 6153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:00.111019 6153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:00.111060 6153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:00.116818 6153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37869
I20260621 05:16:00.116883 6281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37869 every 8 connection(s)
I20260621 05:16:00.117182 6153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:16:00.118413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 6153
I20260621 05:16:00.118494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:16:00.118664 392 external_mini_cluster.cc:1428] Setting key 711456a858b0788ed73454191bc8bf90
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260621 05:16:00.127153 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:00.117875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36623 (local address 127.0.98.62:44217)
0621 05:16:00.117916 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:00.117918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:00.118125 (+ 207us) server_negotiation.cc:407] Connection header received
0621 05:16:00.119096 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:00.119097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:00.119107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:00.119174 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:00.119738 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.120207 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.121717 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.121921 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.121982 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:00.124245 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:00.124246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:00.124246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:00.124253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:00.125805 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.126190 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.126191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.126192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.126249 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.126473 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.126474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.126475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.126611 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:00.126686 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:00.126948 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 05:16:00.127029 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:16:00.127269 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:00.117798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:36623)
0621 05:16:00.117980 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:00.118015 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:00.119006 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:00.119222 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:00.119237 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:00.119452 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:00.119614 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:00.119617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.120307 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.120308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:00.121627 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:00.121630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.122004 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.122005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:00.122155 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:00.122599 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:00.122608 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:00.124151 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:00.125901 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:00.125907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:00.125909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:00.126115 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:00.126299 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:00.126300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:00.126300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:00.126401 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:00.126677 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:00.126683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:00.126814 (+ 131us) client_negotiation.cc:769] Sending connection context
0621 05:16:00.126927 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 05:16:00.127069 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260621 05:16:00.127763 6282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260621 05:16:00.127864 6282 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:00.128011 6282 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260621 05:16:00.128414 5956 ts_manager.cc:194] Registered new tserver with Master: a02e17fd5c414ba1b4d2ac2d199db629 (127.0.98.2:37869)
I20260621 05:16:00.128866 5956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:16:00.140483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:00.220489 6287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:00.220659 6287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:00.220679 6287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:00.221688 6287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:00.221719 6287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:00.221745 6287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:00.223029 6287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:00.223191 6287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:00.223340 6287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:00.227975 6292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:00.227996 6287 server_base.cc:1034] running on GCE node
W20260621 05:16:00.228001 6293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:00.228118 6295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:00.228302 6287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:00.228513 6287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:00.229652 6287 hybrid_clock.cc:648] HybridClock initialized: now 1782018960229642 us; error 43 us; skew 500 ppm
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.232124 6287 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:16:00.232481 6287 webserver.cc:460] Webserver started at http://127.0.98.3:46459/ using document root <none> and password file <none>
I20260621 05:16:00.232683 6287 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:00.232743 6287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:00.232872 6287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:00.233788 6287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "4967735f02d748fc98c3e677347f2603"
format_stamp: "Formatted at 2026-06-21 05:16:00 on dist-test-slave-czm5"
server_key: "780f979c9c2cda697e3dd93aa661b605"
server_key_iv: "f4c9a481393f5805613359da809e2a81"
server_key_version: "encryptionkey@0"
I20260621 05:16:00.234150 6287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4967735f02d748fc98c3e677347f2603"
format_stamp: "Formatted at 2026-06-21 05:16:00 on dist-test-slave-czm5"
server_key: "780f979c9c2cda697e3dd93aa661b605"
server_key_iv: "f4c9a481393f5805613359da809e2a81"
server_key_version: "encryptionkey@0"
I20260621 05:16:00.235379 6287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:16:00.236132 6302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:00.236301 6287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:00.236392 6287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "4967735f02d748fc98c3e677347f2603"
format_stamp: "Formatted at 2026-06-21 05:16:00 on dist-test-slave-czm5"
server_key: "780f979c9c2cda697e3dd93aa661b605"
server_key_iv: "f4c9a481393f5805613359da809e2a81"
server_key_version: "encryptionkey@0"
I20260621 05:16:00.236479 6287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:00.249207 6287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:00.251370 6287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:00.251526 6287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:00.251773 6287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:00.252043 6287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:00.252094 6287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:00.252128 6287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:00.252156 6287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:00.257751 6287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44523
I20260621 05:16:00.257831 6415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44523 every 8 connection(s)
I20260621 05:16:00.258101 6287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:00.264698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 6287
I20260621 05:16:00.264775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:16:00.264979 392 external_mini_cluster.cc:1428] Setting key 5225bdb6b606f0435417f3108c4b9c2f
I20260621 05:16:00.267000 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:00.258740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59111 (local address 127.0.98.62:44217)
0621 05:16:00.258818 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:00.258820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:00.258970 (+ 150us) server_negotiation.cc:407] Connection header received
0621 05:16:00.259832 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:00.259834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:00.259842 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:00.259890 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:00.260379 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.260869 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.261762 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.261956 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.262003 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:00.264212 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:00.264214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:00.264214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:00.264220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:00.265609 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.266064 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.266065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.266066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.266124 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.266364 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.266365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.266365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.266477 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:00.266564 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:00.266833 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 05:16:00.266912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 05:16:00.267139 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:00.258707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:59111)
0621 05:16:00.258863 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:00.258892 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:00.259739 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:00.259937 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:00.259940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:00.260150 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:00.260310 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:00.260313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.260942 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.260943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:00.261692 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:00.261693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.262028 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.262029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:00.262141 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:00.262616 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:00.262624 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:00.264107 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:00.265687 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:00.265693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:00.265695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:00.265976 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:00.266187 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:00.266188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:00.266188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:00.266293 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:00.266573 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:00.266579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:00.266701 (+ 122us) client_negotiation.cc:769] Sending connection context
0621 05:16:00.266796 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:16:00.266935 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260621 05:16:00.267508 6416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260621 05:16:00.267616 6416 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:00.267808 6416 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
I20260621 05:16:00.268157 5956 ts_manager.cc:194] Registered new tserver with Master: 4967735f02d748fc98c3e677347f2603 (127.0.98.3:44523)
I20260621 05:16:00.268501 5956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59111
I20260621 05:16:00.276646 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/test-admin.keytab.
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.290575 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 05:16:00.292019 6423 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:00.291313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.0.98.1:34041)
0621 05:16:00.291457 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:00.291466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:00.291473 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:16:00.291492 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:00.291494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:00.291504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:00.291580 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:00.291943 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43198: BlockingRecv error: recv got EOF from 127.0.0.1:43198 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":54,"threads_started":1}
W20260621 05:16:00.292079 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5cc' of type FILE
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.292943 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.293793 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:16:00.301180 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:00.294258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.0.98.1:34041)
0621 05:16:00.294316 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:00.294318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:00.294411 (+ 93us) server_negotiation.cc:407] Connection header received
0621 05:16:00.294495 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:00.294496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:00.294508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:00.294584 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:00.294959 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.295721 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.296388 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.296728 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.296809 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:00.298630 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:00.298637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:00.298637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:00.298644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:00.300016 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.300350 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.300353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.300353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.300426 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.300632 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.300633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.300633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.300749 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:00.300873 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:00.301004 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 05:16:00.301080 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260621 05:16:00.302873 6423 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:00.302298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.0.98.1:34041)
0621 05:16:00.302371 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:00.302372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:00.302396 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:16:00.302463 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:00.302463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:00.302473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:00.302523 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:00.302792 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43218: BlockingRecv error: recv got EOF from 127.0.0.1:43218 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 05:16:00.302953 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5cc' of type FILE
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.303818 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.304593 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:16:00.310263 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:00.304985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.0.98.1:34041)
0621 05:16:00.305017 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:00.305019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:00.305047 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:16:00.305101 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:00.305101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:00.305107 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:00.305155 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:00.305438 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.305843 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.306435 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.306633 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.306679 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:00.308497 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:00.308498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:00.308498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:00.308503 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:00.309491 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.309752 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.309753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.309753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.309793 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.309951 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.309951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.309951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.310021 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:00.310092 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:00.310170 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:16:00.310216 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260621 05:16:00.311710 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018933478224-392-0/krb5kdc/krb5cc' of type FILE
W20260621 05:16:00.311734 6423 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:00.311142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.0.98.1:34041)
0621 05:16:00.311191 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:00.311192 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:00.311219 (+ 27us) server_negotiation.cc:407] Connection header received
0621 05:16:00.311274 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:00.311275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:00.311281 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:00.311319 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:00.311598 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43242: BlockingRecv error: recv got EOF from 127.0.0.1:43242 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.312534 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:00.313244 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:16:00.319131 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:00.313702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.0.98.1:34041)
0621 05:16:00.313768 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:00.313769 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:00.313793 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:16:00.313839 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:00.313840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:00.313846 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:00.313885 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:00.314144 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.314559 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.315145 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:00.315305 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:00.315350 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:00.317085 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:00.317086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:00.317086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:00.317091 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:00.318326 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.318623 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.318624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.318624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.318663 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:00.318800 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:00.318800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:00.318800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:00.318861 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:00.318913 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:00.318970 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 05:16:00.319014 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:16:00.319720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 6018
I20260621 05:16:00.324278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 6153
I20260621 05:16:00.328416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 6287
I20260621 05:16:00.332317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 5924
2026-06-21T05:16:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2092 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[6436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[6436](info): set up 2 sockets
Jun 21 05:16:00 dist-test-slave-czm5 krb5kdc[6436](info): commencing operation
krb5kdc: starting...
W20260621 05:16:01.311573 392 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:16:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260621 05:16:01.413959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:01.491007 6449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:01.491191 6449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:01.491211 6449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:01.492385 6449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:01.492425 6449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:01.492439 6449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:01.492451 6449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:01.492506 6449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:01.493912 6449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:01.494148 6449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:01.494377 6449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:01.499022 6454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:01.499076 6449 server_base.cc:1034] running on GCE node
W20260621 05:16:01.499033 6455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:01.499038 6457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:01.499408 6449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:01.499732 6449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:01.500885 6449 hybrid_clock.cc:648] HybridClock initialized: now 1782018961500882 us; error 34 us; skew 500 ppm
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:01.507086 6449 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260621 05:16:01.507529 6449 webserver.cc:460] Webserver started at http://127.0.98.62:41275/ using document root <none> and password file <none>
I20260621 05:16:01.507772 6449 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:01.507838 6449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:01.508004 6449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:01.508952 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "d4adbac03f52493a9fced4f80ec801ea"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "ca3dad457611ed1990aa2a4438bd496e"
server_key_iv: "434d97d56844af40c24db724c1e2fef2"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.509311 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4adbac03f52493a9fced4f80ec801ea"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "ca3dad457611ed1990aa2a4438bd496e"
server_key_iv: "434d97d56844af40c24db724c1e2fef2"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.510576 6449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:01.511272 6464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.511442 6449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:01.511540 6449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "d4adbac03f52493a9fced4f80ec801ea"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "ca3dad457611ed1990aa2a4438bd496e"
server_key_iv: "434d97d56844af40c24db724c1e2fef2"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.511615 6449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:01.526546 6449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:01.528678 6449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:01.528827 6449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:01.532835 6449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35707
I20260621 05:16:01.532879 6516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35707 every 8 connection(s)
I20260621 05:16:01.533196 6449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:16:01.534054 6517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:01.536157 6517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: Bootstrap starting.
I20260621 05:16:01.536633 6517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:01.536870 6517 log.cc:824] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:01.537449 6517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: No bootstrap required, opened a new log
I20260621 05:16:01.537752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 6449
I20260621 05:16:01.537868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:16:01.538038 392 external_mini_cluster.cc:1428] Setting key e017876f5c3bc733ba80006e12976344
I20260621 05:16:01.539462 6517 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } }
I20260621 05:16:01.539615 6517 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:01.539708 6517 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4adbac03f52493a9fced4f80ec801ea, State: Initialized, Role: FOLLOWER
I20260621 05:16:01.539844 6517 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } }
I20260621 05:16:01.539932 6517 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:16:01.539964 6517 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:16:01.540012 6517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:01.540884 6517 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } }
I20260621 05:16:01.540964 6517 leader_election.cc:302] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4adbac03f52493a9fced4f80ec801ea; no voters:
I20260621 05:16:01.541122 6517 leader_election.cc:288] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:16:01.541162 6522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:01.541316 6522 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 1 LEADER]: Becoming Leader. State: Replica: d4adbac03f52493a9fced4f80ec801ea, State: Running, Role: LEADER
I20260621 05:16:01.541383 6517 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:01.541445 6522 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } }
I20260621 05:16:01.541859 6523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } }
I20260621 05:16:01.541924 6523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:01.542106 6524 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4adbac03f52493a9fced4f80ec801ea. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } }
I20260621 05:16:01.542191 6524 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:01.542407 6530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:16:01.542914 6530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:16:01.544725 6530 catalog_manager.cc:1347] Generated new cluster ID: d8eeae15136146c996c02c438e5762cb
I20260621 05:16:01.544773 6530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:01.549402 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:01.538514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.0.98.62:35707)
0621 05:16:01.538688 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:01.538695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:01.538731 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:16:01.539450 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:01.539455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:01.539498 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:01.539599 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:01.540361 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.541382 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.542390 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.542969 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.543350 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:01.545513 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:01.545517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:01.545517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:01.545527 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:01.547463 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:01.547903 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:01.547906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:01.547906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:01.548025 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:01.548249 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:01.548249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:01.548250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:01.548517 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:01.549001 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:01.549077 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 05:16:01.549209 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260621 05:16:01.564421 6530 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:16:01.565099 6530 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:16:01.570866 6530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: Generated new TSK 0
I20260621 05:16:01.571031 6530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260621 05:16:01.594059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35707
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:01.670799 6543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:01.671008 6543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:01.671044 6543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:01.672117 6543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:01.672178 6543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:01.672219 6543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:16:01.673623 6543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:01.673871 6543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:01.674086 6543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:01.678763 6551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:01.678762 6548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:01.678817 6549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:01.678927 6543 server_base.cc:1034] running on GCE node
I20260621 05:16:01.679212 6543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:01.679512 6543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:01.680655 6543 hybrid_clock.cc:648] HybridClock initialized: now 1782018961680638 us; error 33 us; skew 500 ppm
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:01.682888 6543 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260621 05:16:01.683310 6543 webserver.cc:460] Webserver started at http://127.0.98.1:36481/ using document root <none> and password file <none>
I20260621 05:16:01.683553 6543 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:01.683620 6543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:01.683769 6543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:01.684746 6543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "0116ad5690c3451290d98befa6c39dd3"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "8b516630468d5a2fed825308b4cfe0ee"
server_key_iv: "de1ac4c9266f88d5d5f5f8c5ba08947a"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.685099 6543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0116ad5690c3451290d98befa6c39dd3"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "8b516630468d5a2fed825308b4cfe0ee"
server_key_iv: "de1ac4c9266f88d5d5f5f8c5ba08947a"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.686334 6543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:16:01.687013 6558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.687183 6543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:01.687269 6543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "0116ad5690c3451290d98befa6c39dd3"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "8b516630468d5a2fed825308b4cfe0ee"
server_key_iv: "de1ac4c9266f88d5d5f5f8c5ba08947a"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.687353 6543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:01.705190 6543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:01.706761 6543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:01.706921 6543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:01.707136 6543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:01.707428 6543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:01.707484 6543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.707518 6543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:01.707544 6543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.713099 6543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35573
I20260621 05:16:01.713138 6671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35573 every 8 connection(s)
I20260621 05:16:01.713410 6543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:16:01.718189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 6543
I20260621 05:16:01.718287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:16:01.718468 392 external_mini_cluster.cc:1428] Setting key a17b4c1a6ca77005c7a879229ee5cac4
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:01.722434 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:01.714006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53553 (local address 127.0.98.62:35707)
0621 05:16:01.714046 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:01.714048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:01.714277 (+ 229us) server_negotiation.cc:407] Connection header received
0621 05:16:01.714751 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:01.714752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:01.714767 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:01.714827 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:01.715792 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.716253 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.717227 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.717457 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.717543 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:01.719850 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:01.719852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:01.719852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:01.719862 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:01.721138 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:01.721522 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:01.721524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:01.721524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:01.721585 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:01.721817 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:01.721818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:01.721818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:01.721922 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:01.722001 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:01.722245 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 05:16:01.722348 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:16:01.722582 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:01.713958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:53553)
0621 05:16:01.714152 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:01.714193 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:01.714650 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:01.714891 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:01.714894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:01.715503 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:01.715697 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:01.715701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.716338 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.716339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:01.717135 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:01.717137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.717552 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.717553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:01.717670 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:01.718080 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:01.718090 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:01.719749 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:01.721233 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:01.721239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:01.721241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:01.721438 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:01.721646 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:01.721647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:01.721647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:01.721749 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:01.721992 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:01.721998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:01.722125 (+ 127us) client_negotiation.cc:769] Sending connection context
0621 05:16:01.722221 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:16:01.722372 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260621 05:16:01.723052 6672 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35707
I20260621 05:16:01.723124 6672 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:01.723285 6672 heartbeater.cc:511] Master 127.0.98.62:35707 requested a full tablet report, sending...
I20260621 05:16:01.723803 6481 ts_manager.cc:194] Registered new tserver with Master: 0116ad5690c3451290d98befa6c39dd3 (127.0.98.1:35573)
I20260621 05:16:01.724349 6481 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53553
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260621 05:16:01.740453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35707
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:01.816746 6677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:01.816926 6677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:01.816947 6677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:01.817982 6677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:01.818017 6677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:01.818040 6677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:01.819295 6677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:01.819447 6677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:01.819600 6677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:01.823930 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:01.823920 6685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:01.823964 6683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:01.824077 6677 server_base.cc:1034] running on GCE node
I20260621 05:16:01.824309 6677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:01.824476 6677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:01.825599 6677 hybrid_clock.cc:648] HybridClock initialized: now 1782018961825590 us; error 32 us; skew 500 ppm
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:01.827829 6677 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260621 05:16:01.828178 6677 webserver.cc:460] Webserver started at http://127.0.98.2:37043/ using document root <none> and password file <none>
I20260621 05:16:01.828348 6677 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:01.828385 6677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:01.828472 6677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:01.829334 6677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "48509c6fb70a41598ee146bd822cdb95"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "596fdf13b0a05600cd1b3eea8c9a6e27"
server_key_iv: "c6d449cdf4caeefa4c2044456085684f"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.829608 6677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48509c6fb70a41598ee146bd822cdb95"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "596fdf13b0a05600cd1b3eea8c9a6e27"
server_key_iv: "c6d449cdf4caeefa4c2044456085684f"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.830670 6677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:16:01.831223 6692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.831388 6677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.831501 6677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "48509c6fb70a41598ee146bd822cdb95"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "596fdf13b0a05600cd1b3eea8c9a6e27"
server_key_iv: "c6d449cdf4caeefa4c2044456085684f"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.831568 6677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:01.845497 6677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:01.847149 6677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:01.847293 6677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:01.847500 6677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:01.847806 6677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:01.847841 6677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.847860 6677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:01.847905 6677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.853245 6677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41973
I20260621 05:16:01.853315 6805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41973 every 8 connection(s)
I20260621 05:16:01.853605 6677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:16:01.854622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 6677
I20260621 05:16:01.854728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:16:01.854915 392 external_mini_cluster.cc:1428] Setting key 7345f5399a8a7c2ae73114c0a6b0440d
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260621 05:16:01.863588 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:01.854241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39795 (local address 127.0.98.62:35707)
0621 05:16:01.854305 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:01.854306 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:01.854566 (+ 260us) server_negotiation.cc:407] Connection header received
0621 05:16:01.855146 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:01.855148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:01.855175 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:01.855239 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:01.856163 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.856937 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.858436 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.858612 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.858660 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:01.860890 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:01.860891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:01.860892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:01.860900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:01.862260 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:01.862655 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:01.862657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:01.862657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:01.862711 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:01.862973 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:01.862974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:01.862974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:01.863080 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:01.863148 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:01.863410 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 05:16:01.863501 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:16:01.863844 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:01.854235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:39795)
0621 05:16:01.854415 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:01.854473 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:01.855041 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:01.855298 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:01.855302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:01.855925 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:01.856080 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:01.856084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.857037 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.857038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:01.858357 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:01.858360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.858692 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.858693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:01.858834 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:01.859317 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:01.859328 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:01.860797 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:01.862357 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:01.862363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:01.862365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:01.862574 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:01.862774 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:01.862775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:01.862776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:01.862902 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:01.863154 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:01.863159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:01.863284 (+ 125us) client_negotiation.cc:769] Sending connection context
0621 05:16:01.863378 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 05:16:01.863517 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260621 05:16:01.864271 6806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35707
I20260621 05:16:01.864368 6806 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:01.864522 6806 heartbeater.cc:511] Master 127.0.98.62:35707 requested a full tablet report, sending...
I20260621 05:16:01.864850 6481 ts_manager.cc:194] Registered new tserver with Master: 48509c6fb70a41598ee146bd822cdb95 (127.0.98.2:41973)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260621 05:16:01.865355 6481 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260621 05:16:01.877707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35707
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:01.954448 6811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:01.954638 6811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:01.954660 6811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:01.955749 6811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:01.955785 6811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:01.955832 6811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:01.957319 6811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:01.957554 6811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:01.957777 6811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:01.962471 6819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:01.962587 6811 server_base.cc:1034] running on GCE node
W20260621 05:16:01.962500 6816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:01.962500 6817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:01.962873 6811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:01.963106 6811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:01.964251 6811 hybrid_clock.cc:648] HybridClock initialized: now 1782018961964247 us; error 36 us; skew 500 ppm
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:01.966547 6811 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260621 05:16:01.966946 6811 webserver.cc:460] Webserver started at http://127.0.98.3:43331/ using document root <none> and password file <none>
I20260621 05:16:01.967170 6811 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:01.967234 6811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:01.967381 6811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:01.968423 6811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "271cdfbfe6b6476e94d1b33e282bda2c"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "3fdfba8fb46e2f53be0757af6c433995"
server_key_iv: "3d717dccfec0d91867b4847964e9f374"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.968781 6811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "271cdfbfe6b6476e94d1b33e282bda2c"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "3fdfba8fb46e2f53be0757af6c433995"
server_key_iv: "3d717dccfec0d91867b4847964e9f374"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.970014 6811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:16:01.970679 6826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.970842 6811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:01.970932 6811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "271cdfbfe6b6476e94d1b33e282bda2c"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "3fdfba8fb46e2f53be0757af6c433995"
server_key_iv: "3d717dccfec0d91867b4847964e9f374"
server_key_version: "encryptionkey@0"
I20260621 05:16:01.971012 6811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:01.987891 6811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:01.989852 6811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:01.990010 6811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:01.990288 6811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:01.990579 6811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:01.990633 6811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.990665 6811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:01.990691 6811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:01.996114 6811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38115
I20260621 05:16:01.996202 6939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38115 every 8 connection(s)
I20260621 05:16:01.996505 6811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:16:02.001740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 6811
I20260621 05:16:02.001845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:16:02.002020 392 external_mini_cluster.cc:1428] Setting key 15f590a59e440579942d7d85466913bf
Jun 21 05:16:01 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:02.005079 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:01.997231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59929 (local address 127.0.98.62:35707)
0621 05:16:01.997293 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:01.997295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:01.997511 (+ 216us) server_negotiation.cc:407] Connection header received
0621 05:16:01.997961 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:01.997962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:01.997970 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:01.998024 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:01.998833 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.999287 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.000198 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.000354 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.000400 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.002637 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.002638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.002639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.002647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.004094 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.004413 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.004414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.004414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.004457 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.004654 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.004654 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.004654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.004743 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.004787 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.004970 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.005022 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:16:02.005230 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:01.997212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:59929)
0621 05:16:01.997375 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:01.997415 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:01.997874 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:01.998072 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:01.998075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:01.998587 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:01.998760 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:01.998764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:01.999358 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:01.999359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.000132 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.000134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.000416 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.000417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.000535 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.001022 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.001029 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.002539 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.004158 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.004163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.004164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.004353 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.004508 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.004508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.004509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.004602 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.004791 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.004794 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.004888 (+ 94us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.004958 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.005057 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260621 05:16:02.005724 6940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35707
I20260621 05:16:02.005800 6940 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.005921 6940 heartbeater.cc:511] Master 127.0.98.62:35707 requested a full tablet report, sending...
I20260621 05:16:02.006232 6481 ts_manager.cc:194] Registered new tserver with Master: 271cdfbfe6b6476e94d1b33e282bda2c (127.0.98.3:38115)
I20260621 05:16:02.006687 6481 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59929
I20260621 05:16:02.014082 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260621 05:16:02.038740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45645
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35707,127.0.98.61:45645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:02.117170 6945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:02.117390 6945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:02.117426 6945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:02.118539 6945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:02.118603 6945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:02.118626 6945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:02.118646 6945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:02.118675 6945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:02.120110 6945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35707,127.0.98.61:45645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
W20260621 05:16:02.120239 6945 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35707,127.0.98.61:45645'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 05:16:02.120393 6945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:02.120595 6945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:02.124943 6950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.125137 6945 server_base.cc:1034] running on GCE node
W20260621 05:16:02.124958 6953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.124964 6951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.125468 6945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:02.125752 6945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:02.126901 6945 hybrid_clock.cc:648] HybridClock initialized: now 1782018962126889 us; error 40 us; skew 500 ppm
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:02.129253 6945 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260621 05:16:02.129664 6945 webserver.cc:460] Webserver started at http://127.0.98.61:35279/ using document root <none> and password file <none>
I20260621 05:16:02.129828 6945 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.129865 6945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.129946 6945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:02.130851 6945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data/instance:
uuid: "75a81ca07d054b8b99d42bf8bd453a57"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "bc21377f9b46730671581d61f5480681"
server_key_iv: "aa5141544e16942575a5b2f13ad4494f"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.131139 6945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal/instance:
uuid: "75a81ca07d054b8b99d42bf8bd453a57"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "bc21377f9b46730671581d61f5480681"
server_key_iv: "aa5141544e16942575a5b2f13ad4494f"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.132393 6945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:02.133060 6960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.133247 6945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:02.133313 6945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
uuid: "75a81ca07d054b8b99d42bf8bd453a57"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "bc21377f9b46730671581d61f5480681"
server_key_iv: "aa5141544e16942575a5b2f13ad4494f"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.133380 6945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.144088 6945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:02.145628 6945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:02.145725 6945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:02.149379 6945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45645
I20260621 05:16:02.149411 7012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45645 every 8 connection(s)
I20260621 05:16:02.149747 6945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
I20260621 05:16:02.150673 7013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:02.152294 7013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:16:02.152904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 6945
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260621 05:16:02.163019 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.153097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.0.98.62:35707)
0621 05:16:02.153180 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.153182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.153438 (+ 256us) server_negotiation.cc:407] Connection header received
0621 05:16:02.153946 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.153947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.153959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.154014 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.154827 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.155613 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.156829 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.157124 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.157193 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.159760 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.159761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.159761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.159767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.161585 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.161977 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.161978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.161978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.162046 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.162305 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.162306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.162306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.162441 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.162524 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.162820 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.162907 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 05:16:02.163141 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.153090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:59390)
0621 05:16:02.153278 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.153324 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.153858 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.154041 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.154044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.154512 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.154733 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.154739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.155718 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.155719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.156725 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.156727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.157202 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.157202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.157287 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.157937 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.157946 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.159685 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.161661 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.161666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.161668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.161907 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.162114 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.162115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.162115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.162225 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.162528 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.162533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.162684 (+ 151us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.162796 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.162908 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260621 05:16:02.163326 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.153365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.0.98.61:45645)
0621 05:16:02.153509 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.153512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.153535 (+ 23us) server_negotiation.cc:407] Connection header received
0621 05:16:02.153872 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.153880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.153922 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.154017 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.154754 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.155894 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.156830 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.157280 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.157329 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.160347 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.160354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.160354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.160361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.162178 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.162501 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.162504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.162504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.162559 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.162753 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.162753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.162753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.162923 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.163071 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.163165 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.163233 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260621 05:16:02.163843 7013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260621 05:16:02.169517 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.164142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:51948)
0621 05:16:02.164179 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.164204 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.164245 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.164363 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.164363 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.164487 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.164525 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.164526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.165127 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.165128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.165617 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.165618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.165984 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.165985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.166045 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.166343 (+ 298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.166346 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.167356 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.168646 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.168647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.168648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.168793 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.168976 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.168976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.168977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.169025 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.169238 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.169238 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.169271 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.169332 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.169416 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260621 05:16:02.169565 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.164179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.0.98.61:45645)
0621 05:16:02.164224 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.164226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.164234 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:02.164299 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.164300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.164309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.164360 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.164593 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.165043 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.165701 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.165906 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.165983 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.167434 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.167435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.167436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.167442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.168572 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.168869 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.168870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.168870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.168917 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.169088 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.169089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.169089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.169180 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.169245 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.169348 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.169461 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":32}
I20260621 05:16:02.170642 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Bootstrap starting.
I20260621 05:16:02.171411 7013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:02.171730 7013 log.cc:824] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:02.172403 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: No bootstrap required, opened a new log
I20260621 05:16:02.173816 7013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } }
I20260621 05:16:02.173936 7013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:02.173959 7013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a81ca07d054b8b99d42bf8bd453a57, State: Initialized, Role: FOLLOWER
I20260621 05:16:02.174093 7013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } }
I20260621 05:16:02.174396 7013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:02.174336 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } }
I20260621 05:16:02.174474 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:16:02.175956 7030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:16:02.176038 7030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:16:02.177425 6945 master_runner.cc:428] Detected that this master 75a81ca07d054b8b99d42bf8bd453a57 is joining an existing cluster
I20260621 05:16:02.177471 6945 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45645
I20260621 05:16:02.184989 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.178352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.0.98.62:35707)
0621 05:16:02.178392 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.178394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.178523 (+ 129us) server_negotiation.cc:407] Connection header received
0621 05:16:02.178627 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.178629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.178641 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.178710 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.179037 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.179681 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.180610 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.180914 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.180984 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.181983 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.181984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.181984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.181991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.183715 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.184150 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.184151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.184152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.184207 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.184439 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.184440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.184440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.184561 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.184596 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.184815 (+ 219us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.184886 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:16:02.185007 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.178325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:51964)
0621 05:16:02.178477 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.178485 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.178538 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.178947 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.178948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.179142 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.179219 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.179220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.180098 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.180099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.180803 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.180804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.181256 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.181257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.181321 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.181696 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.181701 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.182100 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.183954 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.183955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.183956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.184177 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.184357 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.184358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.184358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.184413 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.184663 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.184664 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.184694 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.184769 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.184874 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260621 05:16:02.185294 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.178311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:59394)
0621 05:16:02.178464 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.178473 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.178546 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.178720 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.178720 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.178872 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.178947 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.178949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.179742 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.179743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.180513 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.180515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.180986 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.180987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.181044 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.181507 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.181512 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.181884 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.183799 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.183800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.183801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.184066 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.184280 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.184281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.184281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.184363 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.184652 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.184653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.184694 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.185106 (+ 412us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.185236 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260621 05:16:02.185593 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.178811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.0.98.61:45645)
0621 05:16:02.178860 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.178861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.178867 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:16:02.178881 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.178882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.178888 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.178937 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.179297 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.180010 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.180904 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.181183 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.181245 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.182181 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.182183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.182184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.182191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.183884 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.184245 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.184246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.184246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.184296 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.184475 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.184476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.184476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.184592 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.184660 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.184801 (+ 141us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.184945 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":33}
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260621 05:16:02.194375 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.187528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.0.98.61:45645)
0621 05:16:02.187562 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.187564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.187736 (+ 172us) server_negotiation.cc:407] Connection header received
0621 05:16:02.187843 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.187844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.187853 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.187920 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.188269 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.188932 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.189776 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.190067 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.190135 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.192201 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.192202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.192202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.192207 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.193377 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.193766 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.193767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.193767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.193809 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.193974 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.193974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.193974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.194053 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.194105 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.194269 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.194315 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:16:02.194546 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.187477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:51974)
0621 05:16:02.187604 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.187672 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.187744 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.187932 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.187936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.188156 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.188202 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.188204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.189008 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.189008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.189708 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.189710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.190140 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.190140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.190187 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.190666 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.190671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.192099 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.193447 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.193452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.193455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.193691 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.193859 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.193859 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.193860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.193918 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.194114 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.194119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.194167 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.194249 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.194371 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260621 05:16:02.195011 6481 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45645: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45645
}
attrs {
promote: true
}
}
dest_uuid: "d4adbac03f52493a9fced4f80ec801ea"
cas_config_opid_index: -1
I20260621 05:16:02.195295 6481 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } }
I20260621 05:16:02.195791 7040 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 75a81ca07d054b8b99d42bf8bd453a57 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } }
I20260621 05:16:02.195984 6987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:02.196673 6987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Refusing update from remote peer d4adbac03f52493a9fced4f80ec801ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 05:16:02.196708 7040 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45645
I20260621 05:16:02.196866 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } }
I20260621 05:16:02.196913 6945 master.cc:537] Master@127.0.98.61:45645 shutting down...
I20260621 05:16:02.196923 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:02.197011 7040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:16:02.197845 6465 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260621 05:16:02.198256 6465 consensus_peers.cc:487] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea -> Peer 75a81ca07d054b8b99d42bf8bd453a57 (127.0.98.61:45645): Couldn't send request to peer 75a81ca07d054b8b99d42bf8bd453a57. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 05:16:02.199527 6945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:02.199607 6945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:02.199723 6945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: stopping tablet replica
I20260621 05:16:02.212219 6945 master.cc:552] Master@127.0.98.61:45645 shutdown complete.
I20260621 05:16:02.212265 6945 master_runner.cc:305] Clearing existing system tablet
I20260621 05:16:02.212397 6945 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.212442 6945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.213094 6945 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:02.213587 7044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.213723 6945 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.213780 6945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.213824 6945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
uuid: "75a81ca07d054b8b99d42bf8bd453a57"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "bc21377f9b46730671581d61f5480681"
server_key_iv: "aa5141544e16942575a5b2f13ad4494f"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.214680 6945 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:02.216126 6945 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 05:16:02.216178 6945 log.cc:1196] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 05:16:02.216365 6945 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Deleting consensus metadata
I20260621 05:16:02.216482 6945 master_runner.cc:315] Copying system tablet from 127.0.98.62:35707
I20260621 05:16:02.217098 6945 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35707
I20260621 05:16:02.222889 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.217342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:59398)
0621 05:16:02.217487 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.217496 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.217567 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.217705 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.217706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.217860 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.218152 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.218153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.218772 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.218773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.219317 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.219318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.219728 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.219729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.219796 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.220210 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.220214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.220464 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.221900 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.221901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.221902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.222061 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.222263 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.222264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.222265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.222318 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.222552 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.222553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.222600 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.222692 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.222786 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260621 05:16:02.222887 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.217399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.0.98.62:35707)
0621 05:16:02.217436 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.217438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.217552 (+ 114us) server_negotiation.cc:407] Connection header received
0621 05:16:02.217634 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.217634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.217644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.217700 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.218236 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.218697 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.219409 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.219608 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.219727 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.220556 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.220557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.220558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.220564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.221821 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.222143 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.222144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.222145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.222199 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.222384 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.222384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.222385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.222489 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.222560 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.222716 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.222785 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:16:02.223222 6501 tablet_copy_service.cc:140] P d4adbac03f52493a9fced4f80ec801ea: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 75a81ca07d054b8b99d42bf8bd453a57 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59398)
I20260621 05:16:02.223320 6501 tablet_copy_service.cc:161] P d4adbac03f52493a9fced4f80ec801ea: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 75a81ca07d054b8b99d42bf8bd453a57 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59398: session id = 75a81ca07d054b8b99d42bf8bd453a57-00000000000000000000000000000000
I20260621 05:16:02.223941 6501 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 05:16:02.224464 6945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 05:16:02.225246 6945 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: tablet copy: Starting download of 0 data blocks...
I20260621 05:16:02.225382 6945 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: tablet copy: Starting download of 1 WAL segments...
I20260621 05:16:02.226151 6945 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 05:16:02.226824 6501 tablet_copy_service.cc:342] P d4adbac03f52493a9fced4f80ec801ea: Request end of tablet copy session 75a81ca07d054b8b99d42bf8bd453a57-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59398
I20260621 05:16:02.226881 6501 tablet_copy_service.cc:433] P d4adbac03f52493a9fced4f80ec801ea: ending tablet copy session 75a81ca07d054b8b99d42bf8bd453a57-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 75a81ca07d054b8b99d42bf8bd453a57
I20260621 05:16:02.229300 6945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:02.230310 7051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.230320 6945 server_base.cc:1034] running on GCE node
W20260621 05:16:02.230402 7052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.230402 7054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.230616 6945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:02.230756 6945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:02.231879 6945 hybrid_clock.cc:648] HybridClock initialized: now 1782018962231870 us; error 23 us; skew 500 ppm
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:02.233105 6945 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260621 05:16:02.233414 6945 webserver.cc:460] Webserver started at http://127.0.98.61:43821/ using document root <none> and password file <none>
I20260621 05:16:02.233523 6945 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.233577 6945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.234112 6945 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:02.234627 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.234742 6945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.234789 6945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
uuid: "75a81ca07d054b8b99d42bf8bd453a57"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "bc21377f9b46730671581d61f5480681"
server_key_iv: "aa5141544e16942575a5b2f13ad4494f"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.234840 6945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.238476 6945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:02.239884 6945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:02.242857 6945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45645
I20260621 05:16:02.243075 7113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45645 every 8 connection(s)
I20260621 05:16:02.243207 6945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
I20260621 05:16:02.243948 7114 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:16:02.244355 7114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Bootstrap starting.
I20260621 05:16:02.246354 7114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:16:02.246556 7114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: Bootstrap complete.
I20260621 05:16:02.246759 7114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } }
I20260621 05:16:02.246822 7114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 75a81ca07d054b8b99d42bf8bd453a57, State: Initialized, Role: LEARNER
I20260621 05:16:02.246847 7114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } }
I20260621 05:16:02.247089 7114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:02.247119 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } }
I20260621 05:16:02.247210 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:16:02.248389 7128 catalog_manager.cc:1259] Loaded cluster ID: d8eeae15136146c996c02c438e5762cb
I20260621 05:16:02.248435 7128 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: loading cluster ID for follower catalog manager: success
I20260621 05:16:02.249445 7128 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: acquiring CA information for follower catalog manager: success
I20260621 05:16:02.249702 7128 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:16:02.271545 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.264985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.0.98.61:45645)
0621 05:16:02.265096 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.265098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.265109 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:16:02.265132 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.265132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.265143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.265217 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.265712 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.266756 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.267443 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.267795 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.267872 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.268991 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.268992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.268993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.269000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.270533 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.270833 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.270835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.270835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.270895 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.271067 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.271068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.271069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.271189 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.271286 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.271363 (+ 77us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.271437 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260621 05:16:02.294662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40775
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:02.372676 7132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:02.372920 7132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:02.372943 7132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:02.374060 7132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:02.374100 7132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:02.374114 7132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:02.374152 7132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:02.374167 7132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:02.375574 7132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:02.375821 7132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:02.376031 7132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:02.380275 7140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.380281 7137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.380287 7138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.380510 7132 server_base.cc:1034] running on GCE node
I20260621 05:16:02.380716 7132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:02.380980 7132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:02.382135 7132 hybrid_clock.cc:648] HybridClock initialized: now 1782018962382116 us; error 31 us; skew 500 ppm
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:02.384760 7132 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260621 05:16:02.385186 7132 webserver.cc:460] Webserver started at http://127.0.98.60:42965/ using document root <none> and password file <none>
I20260621 05:16:02.385397 7132 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.385460 7132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.385607 7132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:02.386518 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data/instance:
uuid: "d2264b1c16f34807ae14238f6be1570e"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "c19c0a181f789ec2d39358b0cc1f5273"
server_key_iv: "1cff40210fd40709b3235b6a288e03cd"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.386881 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal/instance:
uuid: "d2264b1c16f34807ae14238f6be1570e"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "c19c0a181f789ec2d39358b0cc1f5273"
server_key_iv: "1cff40210fd40709b3235b6a288e03cd"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.388175 7132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:02.388796 7147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.388952 7132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:02.389041 7132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
uuid: "d2264b1c16f34807ae14238f6be1570e"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "c19c0a181f789ec2d39358b0cc1f5273"
server_key_iv: "1cff40210fd40709b3235b6a288e03cd"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.389120 7132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.411052 7132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:02.412680 7132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:02.412817 7132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:02.417217 7132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40775
I20260621 05:16:02.417244 7199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40775 every 8 connection(s)
I20260621 05:16:02.417938 7132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
I20260621 05:16:02.418803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 7132
I20260621 05:16:02.418910 7200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:02.421700 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:02.430727 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.419385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.0.98.60:40775)
0621 05:16:02.419768 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.419777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.419811 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:16:02.420423 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.420431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.420494 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.420643 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.421658 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.423068 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.424074 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.424624 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.424716 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.427564 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.427570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.427570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.427576 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.429335 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.429681 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.429683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.429684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.429766 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.429958 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.429959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.429959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.430162 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.430303 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.430407 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.430526 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":81,"threads_started":1}
I20260621 05:16:02.430778 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.422540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.98.62:35707)
0621 05:16:02.422827 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.422829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.422837 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:02.422860 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.422861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.422870 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.422915 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.423512 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.424182 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.425105 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.425340 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.425396 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.428286 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.428287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.428287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.428294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.429622 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.429993 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.429994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.429995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.430049 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.430238 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.430239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.430240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.430352 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.430421 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.430586 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.430675 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260621 05:16:02.430814 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.422342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:59414)
0621 05:16:02.422594 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.422634 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.422713 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.422956 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.422958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.423335 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.423397 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.423399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.424269 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.424270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.425006 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.425009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.425474 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.425475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.425543 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.426010 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.426019 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.428178 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.429716 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.429722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.429724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.429906 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.430114 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.430115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.430115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.430170 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.430412 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.430418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.430471 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.430559 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.430673 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":65,"threads_started":1}
I20260621 05:16:02.431391 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260621 05:16:02.436872 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.431593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:52008)
0621 05:16:02.431652 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.431662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.431697 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.431826 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.431827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.431949 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.431985 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.431986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.432594 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.432595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.433058 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.433059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.433436 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.433437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.433499 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.433849 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.433852 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.434918 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.436160 (+ 1242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.436161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.436161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.436316 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.436462 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.436463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.436463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.436511 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.436667 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.436667 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.436694 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.436744 (+ 50us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.436819 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 05:16:02.436885 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.431676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.0.98.61:45645)
0621 05:16:02.431722 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.431724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.431732 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:02.431757 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.431758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.431769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.431823 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.432058 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.432515 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.433149 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.433358 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.433435 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.434981 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.434982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.434983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.434988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.436102 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.436377 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.436378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.436378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.436415 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.436546 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.436546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.436546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.436622 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.436663 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.436759 (+ 96us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.436829 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:16:02.437312 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40775 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260621 05:16:02.442878 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.437534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40775 (local address 127.0.0.1:41416)
0621 05:16:02.437582 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.437593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.437636 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.437753 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.437754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.437874 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.437906 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.437907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.438460 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.438461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.438891 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.438892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.439254 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.439255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.439299 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.439590 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.439593 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.440597 (+ 1004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.442080 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.442081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.442082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.442218 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.442371 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.442371 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.442372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.442413 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.442686 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.442686 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.442725 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.442772 (+ 47us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.442831 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:16:02.442950 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.437583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.0.98.60:40775)
0621 05:16:02.437610 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.437612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.437622 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:16:02.437690 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.437691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.437698 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.437750 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.437972 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.438390 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.438981 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.439178 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.439254 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.440669 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.440670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.440671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.440676 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.442021 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.442277 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.442278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.442278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.442321 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.442465 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.442466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.442466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.442558 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.442650 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.442803 (+ 153us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.442871 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":14}
I20260621 05:16:02.444026 7200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Bootstrap starting.
I20260621 05:16:02.444765 7200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:02.445022 7200 log.cc:824] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:02.445631 7200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: No bootstrap required, opened a new log
I20260621 05:16:02.446956 7200 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40775 } }
I20260621 05:16:02.447079 7200 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:02.447124 7200 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2264b1c16f34807ae14238f6be1570e, State: Initialized, Role: FOLLOWER
I20260621 05:16:02.447258 7200 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40775 } }
I20260621 05:16:02.447530 7205 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40775 } } }
I20260621 05:16:02.447600 7200 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:02.447638 7205 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:16:02.449119 7216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:16:02.449172 7216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:16:02.451092 7132 master_runner.cc:428] Detected that this master d2264b1c16f34807ae14238f6be1570e is joining an existing cluster
I20260621 05:16:02.451143 7132 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40775
I20260621 05:16:02.458058 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.452021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.0.98.62:35707)
0621 05:16:02.452055 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.452057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.452197 (+ 140us) server_negotiation.cc:407] Connection header received
0621 05:16:02.452259 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.452260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.452269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.452319 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.452703 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.453277 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.454266 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.454560 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.454629 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.455590 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.455591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.455591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.455597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.456912 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.457253 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.457254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.457255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.457304 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.457513 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.457514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.457514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.457644 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.457718 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.457887 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.457972 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:16:02.458091 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.451983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:59416)
0621 05:16:02.452119 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.452145 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.452209 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.452322 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.452322 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.452547 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.452607 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.452610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.453382 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.453383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.454192 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.454193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.454646 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.454647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.454702 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.455101 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.455105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.455495 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.457002 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.457003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.457003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.457169 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.457373 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.457374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.457374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.457436 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.457725 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.457726 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.457770 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.457848 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.457934 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":97,"threads_started":1}
I20260621 05:16:02.459331 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.452515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.0.98.61:45645)
0621 05:16:02.452547 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.452549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.452624 (+ 75us) server_negotiation.cc:407] Connection header received
0621 05:16:02.452666 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.452667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.452676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.452719 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.453441 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.454107 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.454935 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.455226 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.455269 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.456522 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.456524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.456524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.456530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.458210 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.458639 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.458640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.458640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.458694 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.458873 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.458874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.458874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.458979 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.459017 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.459183 (+ 166us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.459246 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:16:02.459813 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.452222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:52014)
0621 05:16:02.452559 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.452567 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.452619 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.452771 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.452771 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.452911 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.453348 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.453350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.454190 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.454191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.454847 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.454849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.455338 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.455339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.455398 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.455831 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.455835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.456216 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.458283 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.458284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.458284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.458559 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.458759 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.458760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.458760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.458812 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.459057 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.459058 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.459091 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.459599 (+ 508us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.459706 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":71,"threads_started":1}
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260621 05:16:02.482015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 6543
I20260621 05:16:02.486951 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.460279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40775 (local address 127.0.0.1:41444)
0621 05:16:02.460314 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.460324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.460353 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.460591 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.460592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.460735 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.460770 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.460772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.461339 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.461339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.462047 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.462049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.462344 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.462345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.462415 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.462708 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.462712 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.463913 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.485932 (+ 22019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.485933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.485934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.486153 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.486352 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.486353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.486353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.486406 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.486660 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.486661 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.486702 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.486748 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.486847 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260621 05:16:02.487673 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.460350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.98.60:40775)
0621 05:16:02.460456 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.460458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.460464 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:16:02.460500 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.460500 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.460512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.460569 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.460840 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.461256 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.462133 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.462285 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.462325 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.463996 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.463997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.463997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.464002 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.485838 (+ 21836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.486233 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.486234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.486234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.486283 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.486470 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.486471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.486471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.486586 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.487336 (+ 750us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.487387 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.487517 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260621 05:16:02.488397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35573
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35707
--webserver_port=36481
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:16:02.488459 6481 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40775: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d2264b1c16f34807ae14238f6be1570e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40775
}
attrs {
promote: true
}
}
dest_uuid: "d4adbac03f52493a9fced4f80ec801ea"
cas_config_opid_index: 5
I20260621 05:16:02.488883 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.452407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.0.98.60:40775)
0621 05:16:02.452441 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.452442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.458195 (+ 5753us) server_negotiation.cc:407] Connection header received
0621 05:16:02.458278 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.458278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.458285 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.458309 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.458672 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.459312 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.460193 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.460436 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.460472 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.484835 (+ 24363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.484836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.484836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.484842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.486576 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.486923 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.486924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.486924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.486968 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.487130 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.487130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.487131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.487245 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.487291 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.488672 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.488750 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:16:02.489028 6481 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:02.489171 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.452387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40775 (local address 127.0.0.1:41432)
0621 05:16:02.458147 (+ 5760us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.458157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.458215 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.458365 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.458366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.458560 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.458605 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.458606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.459388 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.459388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.460126 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.460128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.460530 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.460530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.460587 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.484263 (+ 23676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.484268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.484735 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.486658 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.486659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.486659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.486857 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.487024 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.487024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.487024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.487076 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.487315 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.487316 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.487349 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.488953 (+ 1604us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.489038 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5751,"thread_start_us":59,"threads_started":1}
I20260621 05:16:02.489548 7040 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d2264b1c16f34807ae14238f6be1570e (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.490473 7040 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40775
I20260621 05:16:02.490541 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.490602 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:02.490777 7174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:02.491668 7174 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 1 FOLLOWER]: Refusing update from remote peer d4adbac03f52493a9fced4f80ec801ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 05:16:02.491966 7132 master.cc:537] Master@127.0.98.60:40775 shutting down...
I20260621 05:16:02.492580 7039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:16:02.493623 6465 consensus_peers.cc:487] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea -> Peer d2264b1c16f34807ae14238f6be1570e (127.0.98.60:40775): Couldn't send request to peer d2264b1c16f34807ae14238f6be1570e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 05:16:02.494506 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.489432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:33862)
0621 05:16:02.489732 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.489743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.489788 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.490001 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.490002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.490003 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:02.490043 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.490045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.490929 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.490930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.492228 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.492232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.494192 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.494193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.494265 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.494286 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.494342 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.494390 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260621 05:16:02.495218 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.489540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.0.98.61:45645)
0621 05:16:02.489579 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.489593 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.489845 (+ 252us) server_negotiation.cc:407] Connection header received
0621 05:16:02.489871 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.489872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.489896 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.489951 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:02.490110 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.490838 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.492444 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.493072 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.493141 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.493210 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.495075 (+ 1865us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.495126 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:16:02.495491 7088 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 LEARNER]: Refusing update from remote peer d4adbac03f52493a9fced4f80ec801ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 05:16:02.495733 7039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 05:16:02.495908 7132 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:02.496001 7132 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:02.496044 7132 tablet_replica.cc:330] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: stopping tablet replica
I20260621 05:16:02.496285 7088 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d2264b1c16f34807ae14238f6be1570e (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.497342 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.497444 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:16:02.497529 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4adbac03f52493a9fced4f80ec801ea. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: true } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.497609 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:16:02.498344 7039 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: attempting to promote NON_VOTER 75a81ca07d054b8b99d42bf8bd453a57 to VOTER
I20260621 05:16:02.498505 7039 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:02.499410 7088 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 LEARNER]: Refusing update from remote peer d4adbac03f52493a9fced4f80ec801ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 05:16:02.499504 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.499391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40775 (local address 127.0.0.1:60096)
0621 05:16:02.499428 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.499460 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260621 05:16:02.499574 6465 consensus_peers.cc:487] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea -> Peer d2264b1c16f34807ae14238f6be1570e (127.0.98.60:40775): Couldn't send request to peer d2264b1c16f34807ae14238f6be1570e. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 05:16:02.499845 7040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260621 05:16:02.500559 7229 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 75a81ca07d054b8b99d42bf8bd453a57 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.500715 7088 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 75a81ca07d054b8b99d42bf8bd453a57 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.501438 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.501499 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:16:02.501523 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.501591 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:02.501667 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4adbac03f52493a9fced4f80ec801ea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.501706 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:02.511020 7132 master.cc:552] Master@127.0.98.60:40775 shutdown complete.
I20260621 05:16:02.511070 7132 master_runner.cc:305] Clearing existing system tablet
I20260621 05:16:02.511188 7132 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.511229 7132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.511870 7132 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:02.512382 7231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.512508 7132 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.512567 7132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.512590 7132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
uuid: "d2264b1c16f34807ae14238f6be1570e"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "c19c0a181f789ec2d39358b0cc1f5273"
server_key_iv: "1cff40210fd40709b3235b6a288e03cd"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.513237 7132 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:02.514673 7132 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 05:16:02.514739 7132 log.cc:1196] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 05:16:02.514914 7132 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Deleting consensus metadata
I20260621 05:16:02.515022 7132 master_runner.cc:315] Copying system tablet from 127.0.98.62:35707
I20260621 05:16:02.515597 7132 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35707
I20260621 05:16:02.526232 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.515919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:50718)
0621 05:16:02.519731 (+ 3812us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.519743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.519814 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.519977 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.519978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.520178 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.520223 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.520226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.521137 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.521138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.521944 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.521947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.522418 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.522419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.522491 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.522969 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.522974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.523338 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.525207 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.525208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.525209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.525404 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.525603 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.525604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.525604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.525662 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.525918 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.525919 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.525962 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.526054 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.526144 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3791,"thread_start_us":61,"threads_started":1}
I20260621 05:16:02.526261 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.515970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.0.98.62:35707)
0621 05:16:02.516008 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.516010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.519819 (+ 3809us) server_negotiation.cc:407] Connection header received
0621 05:16:02.519878 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.519879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.519892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.519961 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.520314 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.521035 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.522051 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.522339 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.522418 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.523456 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.523457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.523458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.523465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.525127 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.525480 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.525481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.525482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.525538 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.525722 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.525723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.525723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.525840 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.525920 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.526100 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.526173 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:16:02.526866 6501 tablet_copy_service.cc:140] P d4adbac03f52493a9fced4f80ec801ea: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d2264b1c16f34807ae14238f6be1570e ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50718)
I20260621 05:16:02.526927 6501 tablet_copy_service.cc:161] P d4adbac03f52493a9fced4f80ec801ea: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d2264b1c16f34807ae14238f6be1570e at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50718: session id = d2264b1c16f34807ae14238f6be1570e-00000000000000000000000000000000
I20260621 05:16:02.527459 6501 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 05:16:02.527922 7132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 05:16:02.529032 7132 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: tablet copy: Starting download of 0 data blocks...
I20260621 05:16:02.529175 7132 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: tablet copy: Starting download of 1 WAL segments...
I20260621 05:16:02.529759 7132 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 05:16:02.530529 6501 tablet_copy_service.cc:342] P d4adbac03f52493a9fced4f80ec801ea: Request end of tablet copy session d2264b1c16f34807ae14238f6be1570e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50718
I20260621 05:16:02.530579 6501 tablet_copy_service.cc:433] P d4adbac03f52493a9fced4f80ec801ea: ending tablet copy session d2264b1c16f34807ae14238f6be1570e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d2264b1c16f34807ae14238f6be1570e
I20260621 05:16:02.535574 7132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:02.536772 7238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.540527 7239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.540647 7241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.540850 7132 server_base.cc:1034] running on GCE node
I20260621 05:16:02.540958 7132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:02.541074 7132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:02.548365 7132 hybrid_clock.cc:648] HybridClock initialized: now 1782018962548356 us; error 27 us; skew 500 ppm
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:02.549803 7132 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260621 05:16:02.550043 7132 webserver.cc:460] Webserver started at http://127.0.98.60:40451/ using document root <none> and password file <none>
I20260621 05:16:02.550163 7132 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.550194 7132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.550895 7132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:02.551411 7248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.551579 7132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.551649 7132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
uuid: "d2264b1c16f34807ae14238f6be1570e"
format_stamp: "Formatted at 2026-06-21 05:16:02 on dist-test-slave-czm5"
server_key: "c19c0a181f789ec2d39358b0cc1f5273"
server_key_iv: "1cff40210fd40709b3235b6a288e03cd"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.551700 7132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.569629 7132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:02.571413 7132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:02.574566 7132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40775
I20260621 05:16:02.574604 7300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40775 every 8 connection(s)
I20260621 05:16:02.574875 7132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
I20260621 05:16:02.575809 7301 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:16:02.576359 7301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Bootstrap starting.
I20260621 05:16:02.579252 7301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:16:02.579458 7301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: Bootstrap complete.
I20260621 05:16:02.579698 7301 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:02.579792 7301 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d2264b1c16f34807ae14238f6be1570e, State: Initialized, Role: LEARNER
I20260621 05:16:02.579824 7301 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:02.579984 7301 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:02.579993 7304 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } } }
I20260621 05:16:02.580204 7304 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e [sys.catalog]: This master's current role is: LEARNER
I20260621 05:16:02.581278 7315 catalog_manager.cc:1259] Loaded cluster ID: d8eeae15136146c996c02c438e5762cb
I20260621 05:16:02.581362 7315 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: loading cluster ID for follower catalog manager: success
I20260621 05:16:02.582471 7315 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: acquiring CA information for follower catalog manager: success
I20260621 05:16:02.582754 7315 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2264b1c16f34807ae14238f6be1570e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 05:16:02.588152 7226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:02.588342 7226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:02.588388 7226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:02.589387 7226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:02.589462 7226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:02.589504 7226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:16:02.590879 7226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36481
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:02.591120 7226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:02.591312 7226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:02.597074 7321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.597064 7323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.597092 7320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.597324 7226 server_base.cc:1034] running on GCE node
I20260621 05:16:02.597581 7226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:02.597842 7226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:02.599004 7226 hybrid_clock.cc:648] HybridClock initialized: now 1782018962598994 us; error 38 us; skew 500 ppm
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:02.601261 7226 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260621 05:16:02.601673 7226 webserver.cc:460] Webserver started at http://127.0.98.1:36481/ using document root <none> and password file <none>
I20260621 05:16:02.601835 7226 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.601872 7226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.603031 7226 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:02.603566 7330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.603739 7226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.603814 7226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "0116ad5690c3451290d98befa6c39dd3"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "8b516630468d5a2fed825308b4cfe0ee"
server_key_iv: "de1ac4c9266f88d5d5f5f8c5ba08947a"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.604086 7226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.625447 7226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:02.627058 7226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:02.627167 7226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:02.627321 7226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:02.627600 7226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:02.627688 7226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.627743 7226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:02.627769 7226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.633508 7226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35573
I20260621 05:16:02.633571 7443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35573 every 8 connection(s)
I20260621 05:16:02.633884 7226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:02.643795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 7226
I20260621 05:16:02.643949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 6677
I20260621 05:16:02.645256 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.634897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54873 (local address 127.0.98.62:35707)
0621 05:16:02.634989 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.634990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.635000 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:16:02.636097 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.636098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.636111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.636166 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.636897 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.637636 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.639248 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.639502 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.639574 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.642148 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.642149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.642149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.642156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.643723 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644197 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.644198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.644198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.644248 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644503 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.644504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.644504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.644672 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.644721 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.645075 (+ 354us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.645157 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 05:16:02.645625 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.634612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.98.1:44485)
0621 05:16:02.634800 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.634845 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.635999 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.636265 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.636266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.636611 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.636791 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.636797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.637696 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.637697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.638613 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.638616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.639011 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.639012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.639136 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.639665 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.639674 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.642131 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.643844 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.643845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.643845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.644563 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.644777 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.644778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.644837 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.645122 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.645123 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.645166 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.645228 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.645337 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":109,"thread_start_us":101,"threads_started":1}
I20260621 05:16:02.645628 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.634612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:54873)
0621 05:16:02.634800 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.634845 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.635998 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.636231 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.636235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.636612 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.636791 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.636797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.637865 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.637866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.639160 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.639162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.640331 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.640332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.640377 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.640743 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.640750 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.641988 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.643804 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.643806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.643806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.644121 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.644307 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.644308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.644419 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.644769 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.644775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.644921 (+ 146us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.645381 (+ 460us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.645462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":36,"thread_start_us":87,"threads_started":1}
I20260621 05:16:02.645984 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.634584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44485 (local address 127.0.98.61:45645)
0621 05:16:02.634649 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.634651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.635060 (+ 409us) server_negotiation.cc:407] Connection header received
0621 05:16:02.636157 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.636158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.636170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.636215 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.636890 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.637641 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.638708 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.638918 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.639002 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.642203 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.642204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.642205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.642211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.643759 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644646 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.644647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.644647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.644705 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644900 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.644901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.644901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.645041 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.645099 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.645758 (+ 659us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.645837 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:16:02.646067 7445 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35707
I20260621 05:16:02.646152 7445 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.646319 7445 heartbeater.cc:511] Master 127.0.98.62:35707 requested a full tablet report, sending...
I20260621 05:16:02.646448 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.634886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40775 (local address 127.0.98.1:54689)
0621 05:16:02.635029 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.635039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.636006 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.636273 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.636274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.636587 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.636790 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.636794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.637636 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.637637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.639160 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.639162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.639715 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.639716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.639783 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.640190 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.640194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.641696 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.643639 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.643645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.643648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.644443 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.644664 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.644665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.644725 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.644996 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.644997 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.645035 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.645116 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.645261 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":67,"thread_start_us":56,"threads_started":1}
I20260621 05:16:02.646600 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.635035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54689 (local address 127.0.98.60:40775)
0621 05:16:02.635144 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.635145 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.635153 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:02.636151 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.636152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.636164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.636220 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.636884 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.637534 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.639257 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.639590 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.639712 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.641786 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.641788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.641788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.641795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.643531 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644525 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.644526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.644527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.644595 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.644793 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.644794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.644794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.644912 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.644971 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.645564 (+ 593us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.645647 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260621 05:16:02.646726 6481 ts_manager.cc:194] Re-registered known tserver with Master: 0116ad5690c3451290d98befa6c39dd3 (127.0.98.1:35573)
I20260621 05:16:02.647151 6481 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54873
I20260621 05:16:02.649757 7446 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45645
I20260621 05:16:02.649801 7446 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.649865 7446 heartbeater.cc:511] Master 127.0.98.61:45645 requested a full tablet report, sending...
I20260621 05:16:02.650162 7078 ts_manager.cc:194] Registered new tserver with Master: 0116ad5690c3451290d98befa6c39dd3 (127.0.98.1:35573)
I20260621 05:16:02.650942 7444 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40775
I20260621 05:16:02.650997 7444 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.651038 7444 heartbeater.cc:511] Master 127.0.98.60:40775 requested a full tablet report, sending...
I20260621 05:16:02.651180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41973
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35707
--webserver_port=37043
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:16:02.651346 7265 ts_manager.cc:194] Registered new tserver with Master: 0116ad5690c3451290d98befa6c39dd3 (127.0.98.1:35573)
W20260621 05:16:02.728245 7452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:02.728413 7452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:02.728435 7452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:02.729491 7452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:02.729532 7452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:02.729583 7452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:02.730934 7452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37043
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:02.731122 7452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:02.731328 7452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:02.736073 7457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.736060 7460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.736287 7452 server_base.cc:1034] running on GCE node
W20260621 05:16:02.736084 7458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.736495 7452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:02.736660 7452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:02.737792 7452 hybrid_clock.cc:648] HybridClock initialized: now 1782018962737780 us; error 27 us; skew 500 ppm
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:02.740085 7452 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260621 05:16:02.740485 7452 webserver.cc:460] Webserver started at http://127.0.98.2:37043/ using document root <none> and password file <none>
I20260621 05:16:02.740689 7452 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.740764 7452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.742139 7452 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:02.742684 7467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.742841 7452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.742925 7452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "48509c6fb70a41598ee146bd822cdb95"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "596fdf13b0a05600cd1b3eea8c9a6e27"
server_key_iv: "c6d449cdf4caeefa4c2044456085684f"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.743201 7452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.756304 7452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:02.757936 7452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:02.758090 7452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:02.758275 7452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:02.758520 7452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:02.758567 7452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.758601 7452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:02.758625 7452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.763999 7452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41973
I20260621 05:16:02.764037 7580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41973 every 8 connection(s)
I20260621 05:16:02.764343 7452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:16:02.765679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 7452
I20260621 05:16:02.765821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 6811
I20260621 05:16:02.771404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38115
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35707
--webserver_port=43331
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260621 05:16:02.775954 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.765436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39729 (local address 127.0.98.60:40775)
0621 05:16:02.765521 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.765523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.765532 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:02.767472 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.767474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.767489 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.767544 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.768043 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.768706 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.769888 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.770202 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.770277 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.772628 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.772629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.772630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.772638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.774294 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.774777 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.774779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.774779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.774838 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.775123 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.775124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.775124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.775266 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.775342 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.775778 (+ 436us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.775858 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 05:16:02.776155 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.765407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46503 (local address 127.0.98.62:35707)
0621 05:16:02.765451 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.765453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.765462 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:02.765893 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.765895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.765908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.765958 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.767752 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.768519 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.769559 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.769756 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.769836 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.772730 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.772731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.772731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.772737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.774839 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.775258 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.775259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.775260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.775313 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.775512 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.775513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.775514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.775692 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.775759 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.775948 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.776024 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:16:02.776808 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.765010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:46503)
0621 05:16:02.765307 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.765327 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.765775 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.766549 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.766553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.767338 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.767643 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.767649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.768634 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.768634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.769428 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.769430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.769844 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.769844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.769963 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.770504 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.770511 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.772634 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.774956 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.774957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.774957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.775158 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.775381 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.775382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.775382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.775433 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.775787 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.775788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.775831 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.776261 (+ 430us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.776547 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":200,"thread_start_us":69,"threads_started":1}
I20260621 05:16:02.777262 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.765011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40775 (local address 127.0.98.2:39729)
0621 05:16:02.765155 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.765192 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.765748 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.767733 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.767734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.767876 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.767925 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.767926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.768779 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.768780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.769760 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.769762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.770276 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.770277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.770336 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.770697 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.770702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.772516 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.774412 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.774419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.774422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.774694 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.774914 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.774915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.774915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.775047 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.775337 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.775344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.775501 (+ 157us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.775598 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.776464 (+ 866us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260621 05:16:02.777285 7582 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35707
I20260621 05:16:02.777369 7582 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.777558 7582 heartbeater.cc:511] Master 127.0.98.62:35707 requested a full tablet report, sending...
I20260621 05:16:02.777694 7581 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40775
I20260621 05:16:02.777734 7581 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.777814 7581 heartbeater.cc:511] Master 127.0.98.60:40775 requested a full tablet report, sending...
I20260621 05:16:02.777951 6481 ts_manager.cc:194] Re-registered known tserver with Master: 48509c6fb70a41598ee146bd822cdb95 (127.0.98.2:41973)
I20260621 05:16:02.778383 6481 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46503
I20260621 05:16:02.778703 7265 ts_manager.cc:194] Registered new tserver with Master: 48509c6fb70a41598ee146bd822cdb95 (127.0.98.2:41973)
I20260621 05:16:02.778800 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.765527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44853 (local address 127.0.98.61:45645)
0621 05:16:02.765563 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.765565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.766675 (+ 1110us) server_negotiation.cc:407] Connection header received
0621 05:16:02.766708 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.766708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.766722 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.766871 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.769879 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.771109 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.772096 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.772404 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.772466 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.775864 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.775865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.775866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.775871 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.777547 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.777961 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.777962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.777963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.778024 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.778240 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.778240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.778241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.778385 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.778457 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.778624 (+ 167us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.778697 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:16:02.779167 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.765256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.98.2:44853)
0621 05:16:02.766590 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.766600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.766659 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.766922 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.766923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.767338 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.767643 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.767649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.771195 (+ 3546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.771195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.772004 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.772007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.772566 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.772567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.772633 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.773026 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.773030 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.775770 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.777628 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.777629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.777630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.777876 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.778099 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.778099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.778100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.778157 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.778459 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.778459 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.778501 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.778876 (+ 375us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.779035 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"mutex_wait_us":51,"thread_start_us":52,"threads_started":1}
I20260621 05:16:02.779659 7583 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45645
I20260621 05:16:02.779719 7583 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.779795 7583 heartbeater.cc:511] Master 127.0.98.61:45645 requested a full tablet report, sending...
I20260621 05:16:02.780100 7078 ts_manager.cc:194] Registered new tserver with Master: 48509c6fb70a41598ee146bd822cdb95 (127.0.98.2:41973)
W20260621 05:16:02.851013 7588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:02.851190 7588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:02.851212 7588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:02.852270 7588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:02.852308 7588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:02.852334 7588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:02.853763 7588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43331
--tserver_master_addrs=127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:02.853937 7588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:02.854095 7588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:02.858740 7596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.858737 7593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:02.858750 7594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:02.858906 7588 server_base.cc:1034] running on GCE node
I20260621 05:16:02.859161 7588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:02.859401 7588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:02.860571 7588 hybrid_clock.cc:648] HybridClock initialized: now 1782018962860570 us; error 37 us; skew 500 ppm
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:02.862865 7588 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260621 05:16:02.863336 7588 webserver.cc:460] Webserver started at http://127.0.98.3:43331/ using document root <none> and password file <none>
I20260621 05:16:02.863564 7588 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:02.863667 7588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:02.864997 7588 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:02.865513 7603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.865685 7588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:02.865774 7588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "271cdfbfe6b6476e94d1b33e282bda2c"
format_stamp: "Formatted at 2026-06-21 05:16:01 on dist-test-slave-czm5"
server_key: "3fdfba8fb46e2f53be0757af6c433995"
server_key_iv: "3d717dccfec0d91867b4847964e9f374"
server_key_version: "encryptionkey@0"
I20260621 05:16:02.866072 7588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:02.886307 7588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:02.888077 7588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:02.888221 7588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:02.888413 7588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:02.888681 7588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:02.888762 7588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.888796 7588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:02.888828 7588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:02.894164 7588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38115
I20260621 05:16:02.894208 7716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38115 every 8 connection(s)
I20260621 05:16:02.894537 7588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:16:02.896703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 7588
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 21 05:16:02 dist-test-slave-czm5 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:02.906702 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.895215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32919 (local address 127.0.98.60:40775)
0621 05:16:02.895262 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.895264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.895463 (+ 199us) server_negotiation.cc:407] Connection header received
0621 05:16:02.896422 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.896423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.896434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.896465 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.897426 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.898184 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.899689 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.899991 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.900056 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.903520 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.903521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.903521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.903527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.905124 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.905621 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.905622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.905623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.905697 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.905999 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.906000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.906001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.906121 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.906164 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.906532 (+ 368us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.906604 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:16:02.907485 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.895152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40775 (local address 127.0.98.3:32919)
0621 05:16:02.895388 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.895396 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.896332 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.896598 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.896602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.897056 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.897320 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.897325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.898308 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.898308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.899589 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.899591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.900114 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.900115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.900260 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.901155 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.901165 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.903441 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.905216 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.905221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.905223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.905542 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.905776 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.905777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.905777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.905907 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.906211 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.906216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.906384 (+ 168us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.906820 (+ 436us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.907235 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":63,"thread_start_us":88,"threads_started":1}
I20260621 05:16:02.908120 7717 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40775
I20260621 05:16:02.908224 7717 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.908412 7717 heartbeater.cc:511] Master 127.0.98.60:40775 requested a full tablet report, sending...
I20260621 05:16:02.908867 7265 ts_manager.cc:194] Registered new tserver with Master: 271cdfbfe6b6476e94d1b33e282bda2c (127.0.98.3:38115)
I20260621 05:16:02.909070 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.895633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56655 (local address 127.0.98.61:45645)
0621 05:16:02.895685 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.895687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.895694 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:16:02.896422 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.896423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.896433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.896465 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.897425 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.898194 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.899720 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.900018 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.900055 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.904436 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.904437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.904438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.904444 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.906442 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.907914 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.907915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.907916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.907972 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.908474 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.908475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.908475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.908592 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.908644 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.908897 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.908970 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:16:02.909343 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.895436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.98.3:56655)
0621 05:16:02.895545 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.895554 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.896332 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.896598 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.896602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.897058 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.897320 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.897325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.898309 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.898310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.899589 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.899591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.900126 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.900127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.900260 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.901751 (+ 1491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.901756 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.904320 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.907281 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.907282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.907282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.907473 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.908053 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.908054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.908055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.908112 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.908686 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.908687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.908726 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.909135 (+ 409us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.909233 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":45,"thread_start_us":58,"threads_started":1}
I20260621 05:16:02.909778 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.897708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.0.98.62:35707)
0621 05:16:02.898222 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.898223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.898230 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:16:02.898252 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.898252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.898264 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.898322 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.899936 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.900661 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.901708 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.901994 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.902056 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.904066 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.904067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.904067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.904074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.907728 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.908106 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.908108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.908108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.908437 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.908676 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.908677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.908677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.909020 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.909105 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.909553 (+ 448us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.909644 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":78,"threads_started":1}
I20260621 05:16:02.909981 7719 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45645
I20260621 05:16:02.910020 7719 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.910089 7719 heartbeater.cc:511] Master 127.0.98.61:45645 requested a full tablet report, sending...
I20260621 05:16:02.910341 7078 ts_manager.cc:194] Registered new tserver with Master: 271cdfbfe6b6476e94d1b33e282bda2c (127.0.98.3:38115)
I20260621 05:16:02.910692 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.897826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.0.98.61:45645)
0621 05:16:02.898035 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.898037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.900832 (+ 2795us) server_negotiation.cc:407] Connection header received
0621 05:16:02.900910 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.900911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.900923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.900988 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.901669 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.902407 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.903434 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.903722 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.903760 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.905142 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.905143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.905144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.905149 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.907207 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.908209 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.908210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.908210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.908258 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.908829 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.908830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.908831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.908957 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.909060 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.910267 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.910346 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":69,"threads_started":1}
I20260621 05:16:02.911504 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.898162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.0.98.60:40775)
0621 05:16:02.899113 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.899115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.901031 (+ 1916us) server_negotiation.cc:407] Connection header received
0621 05:16:02.901054 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.901055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.901067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.901118 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.902410 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.902900 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.904733 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.905064 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.905126 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.907940 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.907941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.907941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.907948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.909756 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.910526 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.910527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.910528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.910581 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.910782 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.910782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.910783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.910928 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.911018 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.911301 (+ 283us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.911380 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":49,"threads_started":1}
I20260621 05:16:02.912920 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.895153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:52705)
0621 05:16:02.895299 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.895336 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.896314 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.898344 (+ 2030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.898345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.898515 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:02.898580 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.898581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.899402 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.899403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.900124 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.900125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.900652 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.900653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.900731 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.904070 (+ 3339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:02.904075 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:02.907142 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:02.912103 (+ 4961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.912103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.912104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.912248 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.912449 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:02.912450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:02.912450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:02.912499 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:02.912727 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:02.912728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:02.912757 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.912798 (+ 41us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.912859 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260621 05:16:02.912981 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.895143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52705 (local address 127.0.98.62:35707)
0621 05:16:02.895201 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.895203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.895434 (+ 231us) server_negotiation.cc:407] Connection header received
0621 05:16:02.896422 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.896424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.896436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.896500 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:02.898681 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.899328 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.900207 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.900519 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.900560 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.909698 (+ 9138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:02.909699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:02.909700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:02.909707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:02.912025 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.912325 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.912326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.912326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.912390 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:02.912561 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:02.912561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:02.912562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:02.912672 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:02.912731 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.912828 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.912897 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:16:02.911305 6481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 05:16:02.913514 7718 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35707
W20260621 05:16:02.913506 6481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:16:02.913554 7718 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:02.913626 7718 heartbeater.cc:511] Master 127.0.98.62:35707 requested a full tablet report, sending...
I20260621 05:16:02.913825 6480 ts_manager.cc:194] Re-registered known tserver with Master: 271cdfbfe6b6476e94d1b33e282bda2c (127.0.98.3:38115)
I20260621 05:16:02.914247 6480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52705
I20260621 05:16:02.920949 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.917040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41973 (local address 127.0.0.1:59748)
0621 05:16:02.917528 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.917537 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.917696 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.917992 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.917993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.917995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:02.918047 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.918048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.919225 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.919226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.920069 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.920072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.920712 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.920713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.920769 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.920774 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.920822 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.920859 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":69,"threads_started":1}
I20260621 05:16:02.921242 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.917358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.0.98.2:41973)
0621 05:16:02.917819 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.917823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.917832 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:02.917871 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.917873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.917889 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.917966 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:02.918121 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.919146 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.920163 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.920616 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.920708 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.921031 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.921146 (+ 115us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.921191 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":73,"threads_started":1}
I20260621 05:16:02.921841 7515 tablet_service.cc:1467] Processing CreateTablet for tablet 4308aa140eed4615ba1941c57ee33e06 (DEFAULT_TABLE table=test-table [id=92cfdb30601a469f823bd2fe3c5dd431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:02.922188 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4308aa140eed4615ba1941c57ee33e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:02.922717 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.916954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35573 (local address 127.0.0.1:36038)
0621 05:16:02.917140 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.917151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.917675 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.919004 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.919005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.919007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:02.919043 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.919045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.920476 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.920476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.921522 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.921524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.922514 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.922515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.922570 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.922574 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.922632 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.922663 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260621 05:16:02.922732 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.917029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.0.98.1:35573)
0621 05:16:02.917205 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.917210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.918792 (+ 1582us) server_negotiation.cc:407] Connection header received
0621 05:16:02.918838 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.918840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.918855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.918941 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:02.919293 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.920390 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.921668 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.922349 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.922424 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.922482 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.922647 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.922679 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260621 05:16:02.924139 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.917194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.0.98.3:38115)
0621 05:16:02.918087 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:02.918091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:02.918100 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:02.918132 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:02.918134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:02.918148 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:02.918248 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:02.918433 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.919419 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.920689 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.923829 (+ 3140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.923900 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.923958 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:02.924049 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 05:16:02.924078 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":467,"threads_started":1}
I20260621 05:16:02.924132 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:02.917012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38115 (local address 127.0.0.1:55482)
0621 05:16:02.917140 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:02.917149 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:02.917189 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:02.918265 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:02.918265 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:02.918267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:02.918294 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.918295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.919502 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.919503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.920607 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:02.920609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:02.923899 (+ 3290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:02.923900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:02.923953 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:02.923957 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:16:02.924036 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 05:16:02.924071 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260621 05:16:02.924229 7378 tablet_service.cc:1467] Processing CreateTablet for tablet 4308aa140eed4615ba1941c57ee33e06 (DEFAULT_TABLE table=test-table [id=92cfdb30601a469f823bd2fe3c5dd431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:02.924531 7378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4308aa140eed4615ba1941c57ee33e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:02.925240 7743 tablet_bootstrap.cc:492] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Bootstrap starting.
I20260621 05:16:02.925321 7651 tablet_service.cc:1467] Processing CreateTablet for tablet 4308aa140eed4615ba1941c57ee33e06 (DEFAULT_TABLE table=test-table [id=92cfdb30601a469f823bd2fe3c5dd431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:02.925607 7651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4308aa140eed4615ba1941c57ee33e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:02.925840 7743 tablet_bootstrap.cc:654] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:02.926105 7743 log.cc:824] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:02.926864 7743 tablet_bootstrap.cc:492] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: No bootstrap required, opened a new log
I20260621 05:16:02.926921 7743 ts_tablet_manager.cc:1397] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:16:02.927227 7745 tablet_bootstrap.cc:492] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: Bootstrap starting.
I20260621 05:16:02.927862 7745 tablet_bootstrap.cc:654] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:02.928074 7745 log.cc:824] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:02.928704 7745 tablet_bootstrap.cc:492] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: No bootstrap required, opened a new log
I20260621 05:16:02.928756 7745 ts_tablet_manager.cc:1397] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:16:02.928810 7747 tablet_bootstrap.cc:492] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: Bootstrap starting.
I20260621 05:16:02.928913 7743 raft_consensus.cc:348] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:02.929098 7743 raft_consensus.cc:374] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:02.929145 7743 raft_consensus.cc:729] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48509c6fb70a41598ee146bd822cdb95, State: Initialized, Role: FOLLOWER
I20260621 05:16:02.929248 7743 consensus_queue.cc:260] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:02.929414 7747 tablet_bootstrap.cc:654] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:02.929489 7743 ts_tablet_manager.cc:1428] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:16:02.929598 7582 heartbeater.cc:503] Master 127.0.98.62:35707 was elected leader, sending a full tablet report...
I20260621 05:16:02.931100 7747 log.cc:824] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:02.931780 7745 raft_consensus.cc:348] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:02.931968 7745 raft_consensus.cc:374] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:02.932003 7745 raft_consensus.cc:729] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0116ad5690c3451290d98befa6c39dd3, State: Initialized, Role: FOLLOWER
I20260621 05:16:02.932076 7747 tablet_bootstrap.cc:492] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: No bootstrap required, opened a new log
I20260621 05:16:02.932137 7747 ts_tablet_manager.cc:1397] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 05:16:02.932112 7745 consensus_queue.cc:260] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:02.932343 7745 ts_tablet_manager.cc:1428] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 05:16:02.932380 7445 heartbeater.cc:503] Master 127.0.98.62:35707 was elected leader, sending a full tablet report...
I20260621 05:16:02.933712 7747 raft_consensus.cc:348] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:02.933821 7747 raft_consensus.cc:374] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:02.933840 7747 raft_consensus.cc:729] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271cdfbfe6b6476e94d1b33e282bda2c, State: Initialized, Role: FOLLOWER
I20260621 05:16:02.933967 7747 consensus_queue.cc:260] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:02.934237 7718 heartbeater.cc:503] Master 127.0.98.62:35707 was elected leader, sending a full tablet report...
I20260621 05:16:02.934815 7747 ts_tablet_manager.cc:1428] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260621 05:16:03.015410 7584 tablet.cc:2369] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:16:03.025610 7750 raft_consensus.cc:482] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:16:03.025755 7750 raft_consensus.cc:504] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:03.026099 7750 leader_election.cc:288] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 271cdfbfe6b6476e94d1b33e282bda2c (127.0.98.3:38115), 48509c6fb70a41598ee146bd822cdb95 (127.0.98.2:41973)
I20260621 05:16:03.029018 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:03.026358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55677 (local address 127.0.98.2:41973)
0621 05:16:03.026427 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:03.026428 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:03.026436 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:03.026458 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:03.026459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:03.026468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:03.026520 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:03.026690 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.027223 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.028322 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.028666 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.028755 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:03.028765 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:03.028913 (+ 148us) server_negotiation.cc:299] Negotiation successful
0621 05:16:03.028948 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:16:03.029018 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:03.026348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47613 (local address 127.0.98.3:38115)
0621 05:16:03.026522 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:03.026525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:03.026533 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:03.026562 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:03.026563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:03.026574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:03.026629 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:03.026798 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.027287 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.028283 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.028665 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.028756 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:03.028767 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:03.028916 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 05:16:03.028951 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260621 05:16:03.029028 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:03.026277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41973 (local address 127.0.98.1:55677)
0621 05:16:03.026320 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:03.026332 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:03.026407 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:03.026549 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:03.026550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:03.026551 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:03.026622 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:03.026624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.027307 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.027308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:03.028202 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:03.028205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.028755 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.028756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:03.028832 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:03.028834 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:16:03.028907 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:16:03.028946 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":50}
I20260621 05:16:03.029038 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:03.026281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38115 (local address 127.0.98.1:47613)
0621 05:16:03.026368 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:03.026377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:03.026406 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:03.026651 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:03.026669 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:03.026671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:03.026729 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:03.026731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.027371 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.027372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:03.028195 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:03.028197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.028756 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.028757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:03.028834 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:03.028837 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:16:03.028910 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:16:03.028948 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":60}
I20260621 05:16:03.029363 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4308aa140eed4615ba1941c57ee33e06" candidate_uuid: "0116ad5690c3451290d98befa6c39dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48509c6fb70a41598ee146bd822cdb95" is_pre_election: true
I20260621 05:16:03.029520 7535 raft_consensus.cc:2454] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0116ad5690c3451290d98befa6c39dd3 in term 0.
I20260621 05:16:03.029757 7332 leader_election.cc:302] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0116ad5690c3451290d98befa6c39dd3, 48509c6fb70a41598ee146bd822cdb95; no voters:
I20260621 05:16:03.029732 7671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4308aa140eed4615ba1941c57ee33e06" candidate_uuid: "0116ad5690c3451290d98befa6c39dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" is_pre_election: true
I20260621 05:16:03.029867 7671 raft_consensus.cc:2454] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0116ad5690c3451290d98befa6c39dd3 in term 0.
I20260621 05:16:03.029986 7750 raft_consensus.cc:2781] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:16:03.030067 7750 raft_consensus.cc:482] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:16:03.030092 7750 raft_consensus.cc:3037] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:03.030742 7750 raft_consensus.cc:504] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:03.030879 7750 leader_election.cc:288] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [CANDIDATE]: Term 1 election: Requested vote from peers 271cdfbfe6b6476e94d1b33e282bda2c (127.0.98.3:38115), 48509c6fb70a41598ee146bd822cdb95 (127.0.98.2:41973)
I20260621 05:16:03.031026 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4308aa140eed4615ba1941c57ee33e06" candidate_uuid: "0116ad5690c3451290d98befa6c39dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48509c6fb70a41598ee146bd822cdb95"
I20260621 05:16:03.031028 7671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4308aa140eed4615ba1941c57ee33e06" candidate_uuid: "0116ad5690c3451290d98befa6c39dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271cdfbfe6b6476e94d1b33e282bda2c"
I20260621 05:16:03.031096 7671 raft_consensus.cc:3037] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:03.031119 7535 raft_consensus.cc:3037] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:03.031826 7535 raft_consensus.cc:2454] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0116ad5690c3451290d98befa6c39dd3 in term 1.
I20260621 05:16:03.031847 7671 raft_consensus.cc:2454] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0116ad5690c3451290d98befa6c39dd3 in term 1.
I20260621 05:16:03.031993 7332 leader_election.cc:302] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0116ad5690c3451290d98befa6c39dd3, 48509c6fb70a41598ee146bd822cdb95; no voters:
I20260621 05:16:03.032083 7750 raft_consensus.cc:2781] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:03.032228 7750 raft_consensus.cc:686] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 0116ad5690c3451290d98befa6c39dd3, State: Running, Role: LEADER
I20260621 05:16:03.032426 7750 consensus_queue.cc:237] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } }
I20260621 05:16:03.033205 6480 catalog_manager.cc:5495] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0116ad5690c3451290d98befa6c39dd3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0116ad5690c3451290d98befa6c39dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0116ad5690c3451290d98befa6c39dd3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 } health_report { overall_health: UNKNOWN } } }
I20260621 05:16:03.066844 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:03.064533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.0.98.1:35573)
0621 05:16:03.064604 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:03.064606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:03.064614 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:03.064670 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:03.064671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:03.064690 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:03.064760 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:03.064897 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.065360 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.066084 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.066293 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.066367 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:03.066525 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:03.066627 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:03.066719 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:03.066744 (+ 25us) server_negotiation.cc:299] Negotiation successful
0621 05:16:03.066761 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:16:03.068490 7535 raft_consensus.cc:1261] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 1 FOLLOWER]: Refusing update from remote peer 0116ad5690c3451290d98befa6c39dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:03.068503 7671 raft_consensus.cc:1261] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 1 FOLLOWER]: Refusing update from remote peer 0116ad5690c3451290d98befa6c39dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:03.068838 7752 consensus_queue.cc:1035] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48509c6fb70a41598ee146bd822cdb95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:03.068994 7750 consensus_queue.cc:1035] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271cdfbfe6b6476e94d1b33e282bda2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:03.071147 7755 mvcc.cc:205] Tried to move back new op lower bound from 7299149672726003712 to 7299149672581459968. Current Snapshot: MvccSnapshot[applied={T|T < 7299149672726003712}]
I20260621 05:16:03.071898 7757 mvcc.cc:205] Tried to move back new op lower bound from 7299149672726003712 to 7299149672581459968. Current Snapshot: MvccSnapshot[applied={T|T < 7299149672726003712}]
I20260621 05:16:03.072779 7756 mvcc.cc:205] Tried to move back new op lower bound from 7299149672726003712 to 7299149672581459968. Current Snapshot: MvccSnapshot[applied={T|T < 7299149672726003712}]
I20260621 05:16:03.074306 6480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:03.074425 6480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:03.075894 6480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4adbac03f52493a9fced4f80ec801ea: Sending DeleteTablet for 3 replicas of tablet 4308aa140eed4615ba1941c57ee33e06
I20260621 05:16:03.076192 7651 tablet_service.cc:1514] Processing DeleteTablet for tablet 4308aa140eed4615ba1941c57ee33e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55482
I20260621 05:16:03.076196 7378 tablet_service.cc:1514] Processing DeleteTablet for tablet 4308aa140eed4615ba1941c57ee33e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36038
I20260621 05:16:03.076399 7377 tablet_service.cc:1514] Processing DeleteTablet for tablet 4308aa140eed4615ba1941c57ee33e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36038
I20260621 05:16:03.076424 7765 tablet_replica.cc:330] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: stopping tablet replica
I20260621 05:16:03.076512 7765 raft_consensus.cc:2229] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:03.076498 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 4308aa140eed4615ba1941c57ee33e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59748
I20260621 05:16:03.076608 7765 raft_consensus.cc:2258] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:03.076884 7766 tablet_replica.cc:330] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: stopping tablet replica
I20260621 05:16:03.077023 7766 raft_consensus.cc:2229] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:16:03.077098 7765 ts_tablet_manager.cc:1905] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:03.077214 7766 raft_consensus.cc:2258] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:03.077690 7766 ts_tablet_manager.cc:1905] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:03.077728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 6449
I20260621 05:16:03.080003 7766 ts_tablet_manager.cc:1918] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:03.080132 7766 log.cc:1196] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/4308aa140eed4615ba1941c57ee33e06
I20260621 05:16:03.080646 7766 ts_tablet_manager.cc:1939] T 4308aa140eed4615ba1941c57ee33e06 P 0116ad5690c3451290d98befa6c39dd3: Deleting consensus metadata
I20260621 05:16:03.083719 7765 ts_tablet_manager.cc:1918] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:03.084194 7765 log.cc:1196] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/4308aa140eed4615ba1941c57ee33e06
W20260621 05:16:03.084944 7331 connection.cc:535] server connection from 127.0.0.1:36038 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:16:03.085251 7765 ts_tablet_manager.cc:1939] T 4308aa140eed4615ba1941c57ee33e06 P 271cdfbfe6b6476e94d1b33e282bda2c: Deleting consensus metadata
W20260621 05:16:03.085654 7606 connection.cc:441] server connection from 127.0.0.1:55482 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55482 (request call id 1) could send its response
I20260621 05:16:03.085696 7767 tablet_replica.cc:330] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: stopping tablet replica
I20260621 05:16:03.085923 7767 raft_consensus.cc:2229] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:03.086084 7767 raft_consensus.cc:2258] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:03.086349 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:03.077081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.98.60:40775)
0621 05:16:03.077119 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:03.077120 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:03.079307 (+ 2187us) server_negotiation.cc:407] Connection header received
0621 05:16:03.079404 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:03.079405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:03.079417 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:03.079445 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:03.080301 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.081091 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.081810 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.082027 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.082097 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:03.083352 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:03.083353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:03.083353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:03.083360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:03.084903 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:03.085471 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:03.085472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:03.085472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:03.085524 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:03.085754 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:03.085755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:03.085755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:03.085887 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:03.085962 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:03.086167 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 05:16:03.086251 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:16:03.086761 7767 ts_tablet_manager.cc:1905] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:03.088495 7767 ts_tablet_manager.cc:1918] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:03.088603 7767 log.cc:1196] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/4308aa140eed4615ba1941c57ee33e06
I20260621 05:16:03.088878 7767 ts_tablet_manager.cc:1939] T 4308aa140eed4615ba1941c57ee33e06 P 48509c6fb70a41598ee146bd822cdb95: Deleting consensus metadata
W20260621 05:16:03.089229 7470 connection.cc:441] server connection from 127.0.0.1:59748 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59748 (request call id 1) could send its response
I20260621 05:16:03.094942 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:03.086670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.0.98.61:45645)
0621 05:16:03.086709 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:03.086710 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:03.087179 (+ 469us) server_negotiation.cc:407] Connection header received
0621 05:16:03.087270 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:03.087271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:03.087282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:03.087312 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:03.087981 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.088728 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.089870 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.090055 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.090129 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:03.091502 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:03.091504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:03.091504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:03.091511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:03.093147 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:03.093989 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:03.093990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:03.093990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:03.094067 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:03.094348 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:03.094349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:03.094349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:03.094498 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:03.094592 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:03.094684 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 05:16:03.094763 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:16:03.095041 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:03.086957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.98.60:40775)
0621 05:16:03.086988 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:03.086989 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:03.087283 (+ 294us) server_negotiation.cc:407] Connection header received
0621 05:16:03.087371 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:03.087372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:03.087380 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:03.087421 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:03.088123 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.088783 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.090675 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:03.090974 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:03.091052 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:03.092573 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:03.092574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:03.092575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:03.092581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:03.093921 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:03.094267 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:03.094268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:03.094268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:03.094318 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:03.094511 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:03.094512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:03.094512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:03.094624 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:03.094703 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:03.094871 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 05:16:03.094951 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:16:04.073242 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:04.072948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:52953)
0621 05:16:04.073082 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:04.073171 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
W20260621 05:16:04.073536 7582 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:04.074745 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:04.074496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:47483)
0621 05:16:04.074604 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:04.074680 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":47,"threads_started":1}
W20260621 05:16:04.074923 7718 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:04.077656 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:04.077466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:50935)
0621 05:16:04.077570 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:04.077623 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":47,"threads_started":1}
W20260621 05:16:04.077843 7445 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:04.576576 7779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4adbac03f52493a9fced4f80ec801ea)
I20260621 05:16:04.576686 7779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:04.576864 7779 leader_election.cc:288] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707)
I20260621 05:16:04.577172 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:04.576981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:51184)
0621 05:16:04.577093 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:04.577140 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
W20260621 05:16:04.577299 7064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:04.577656 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:04.577565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:51186)
0621 05:16:04.577612 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:04.577637 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:16:04.577746 7064 leader_election.cc:334] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:04.577821 7064 leader_election.cc:302] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 75a81ca07d054b8b99d42bf8bd453a57; no voters: d4adbac03f52493a9fced4f80ec801ea
I20260621 05:16:04.577912 7779 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:16:05.074100 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:05.073881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:45253)
0621 05:16:05.073999 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:05.074040 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260621 05:16:05.075356 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:05.075191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:58799)
0621 05:16:05.075288 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:05.075323 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
I20260621 05:16:05.078307 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:05.078122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:59195)
0621 05:16:05.078237 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:05.078274 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
I20260621 05:16:06.074898 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:06.074656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:44189)
0621 05:16:06.074776 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:06.074839 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260621 05:16:06.075944 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:06.075787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:52029)
0621 05:16:06.075878 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:06.075911 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":43,"threads_started":1}
I20260621 05:16:06.078804 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:06.078634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:56725)
0621 05:16:06.078745 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:06.078771 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260621 05:16:06.225435 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4adbac03f52493a9fced4f80ec801ea)
I20260621 05:16:06.225550 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:06.225726 7788 leader_election.cc:288] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707)
I20260621 05:16:06.226069 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:06.225896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:51394)
0621 05:16:06.225987 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:06.226016 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
I20260621 05:16:06.226478 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:06.226364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:51398)
0621 05:16:06.226412 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:06.226459 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:16:06.226555 7064 leader_election.cc:334] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:06.226624 7064 leader_election.cc:302] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 75a81ca07d054b8b99d42bf8bd453a57; no voters: d4adbac03f52493a9fced4f80ec801ea
I20260621 05:16:06.226712 7788 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:16:07.075672 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:07.075353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:59965)
0621 05:16:07.075478 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:07.075577 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260621 05:16:07.076506 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:07.076339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:50653)
0621 05:16:07.076437 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:07.076468 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
I20260621 05:16:07.079310 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:07.079144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:41675)
0621 05:16:07.079244 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:07.079277 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":44,"threads_started":1}
I20260621 05:16:08.076277 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:08.076066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:57125)
0621 05:16:08.076185 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:08.076224 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260621 05:16:08.076987 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:08.076826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:46775)
0621 05:16:08.076920 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:08.076955 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":60,"threads_started":1}
I20260621 05:16:08.079897 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:08.079764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:47199)
0621 05:16:08.079845 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:08.079866 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":40,"threads_started":1}
I20260621 05:16:08.615481 7797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4adbac03f52493a9fced4f80ec801ea)
I20260621 05:16:08.615566 7797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:08.615795 7797 leader_election.cc:288] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707)
I20260621 05:16:08.616134 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:08.615948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:51642)
0621 05:16:08.616041 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:08.616086 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":52,"threads_started":1}
I20260621 05:16:08.616472 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:08.616394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:51646)
0621 05:16:08.616434 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:08.616457 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 05:16:08.616541 7064 leader_election.cc:334] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:08.616612 7064 leader_election.cc:302] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 75a81ca07d054b8b99d42bf8bd453a57; no voters: d4adbac03f52493a9fced4f80ec801ea
I20260621 05:16:08.616688 7797 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:16:09.077092 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:09.076826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:37541)
0621 05:16:09.077005 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:09.077039 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260621 05:16:09.077487 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:09.077305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:54909)
0621 05:16:09.077391 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:09.077419 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":47,"threads_started":1}
I20260621 05:16:09.080636 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:09.080478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:52595)
0621 05:16:09.080576 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:09.080603 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":56,"threads_started":1}
I20260621 05:16:10.078075 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:10.077878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:40521)
0621 05:16:10.077998 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:10.078033 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":49,"threads_started":1}
I20260621 05:16:10.078075 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:10.077709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:46773)
0621 05:16:10.077999 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:10.078032 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":60,"threads_started":1}
I20260621 05:16:10.081166 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:10.081030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:52323)
0621 05:16:10.081114 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:10.081135 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":44,"threads_started":1}
I20260621 05:16:10.675257 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4adbac03f52493a9fced4f80ec801ea)
I20260621 05:16:10.675339 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:10.675470 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707)
I20260621 05:16:10.675809 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:10.675608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:51792)
0621 05:16:10.675739 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:10.675775 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
W20260621 05:16:10.675894 7064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260621 05:16:10.676239 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:10.676113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:51802)
0621 05:16:10.676159 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:10.676194 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260621 05:16:10.676312 7064 leader_election.cc:334] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:10.676354 7064 leader_election.cc:302] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 75a81ca07d054b8b99d42bf8bd453a57; no voters: d4adbac03f52493a9fced4f80ec801ea
I20260621 05:16:10.676415 7806 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:16:11.078701 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:11.078472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:43233)
0621 05:16:11.078604 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:11.078644 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260621 05:16:11.078763 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:11.078594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:36403)
0621 05:16:11.078697 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:11.078723 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":45,"threads_started":1}
I20260621 05:16:11.081655 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:11.081481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:33777)
0621 05:16:11.081591 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:11.081622 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
I20260621 05:16:12.079469 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:12.079239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:57441)
0621 05:16:12.079389 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:12.079419 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260621 05:16:12.079469 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:12.079238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:43393)
0621 05:16:12.079380 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:12.079420 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260621 05:16:12.082156 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:12.081989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:34083)
0621 05:16:12.082094 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:12.082121 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":58,"threads_started":1}
I20260621 05:16:13.080159 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:13.079959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:58735)
0621 05:16:13.080063 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:13.080106 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":38,"threads_started":1}
I20260621 05:16:13.080346 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:13.079834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:48441)
0621 05:16:13.080274 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:13.080308 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":73,"threads_started":1}
I20260621 05:16:13.082659 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:13.082482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:52239)
0621 05:16:13.082595 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:13.082627 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260621 05:16:13.459744 7818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4adbac03f52493a9fced4f80ec801ea)
I20260621 05:16:13.459849 7818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:13.460026 7818 leader_election.cc:288] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707)
I20260621 05:16:13.460350 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:13.460179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:37884)
0621 05:16:13.460287 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:13.460316 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260621 05:16:13.460729 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:13.460632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:37896)
0621 05:16:13.460679 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:13.460710 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260621 05:16:13.460829 7064 leader_election.cc:334] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:13.460880 7064 leader_election.cc:302] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 75a81ca07d054b8b99d42bf8bd453a57; no voters: d4adbac03f52493a9fced4f80ec801ea
I20260621 05:16:13.460954 7818 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:16:14.081105 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:14.080638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:40093)
0621 05:16:14.080912 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:14.080952 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":59,"threads_started":1}
I20260621 05:16:14.081153 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:14.080772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:52835)
0621 05:16:14.081079 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:14.081111 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":55,"threads_started":1}
I20260621 05:16:14.083274 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:14.083103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:56829)
0621 05:16:14.083208 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:14.083241 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
I20260621 05:16:15.081923 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:15.081632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:59721)
0621 05:16:15.081821 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:15.081861 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":62,"threads_started":1}
I20260621 05:16:15.081974 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:15.081719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:41355)
0621 05:16:15.081886 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:15.081922 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":52,"threads_started":1}
I20260621 05:16:15.083870 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:15.083680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:42347)
0621 05:16:15.083803 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:15.083837 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260621 05:16:16.082537 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:16.082298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:51467)
0621 05:16:16.082443 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:16.082482 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260621 05:16:16.082578 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:16.082398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:60259)
0621 05:16:16.082508 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:16.082537 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260621 05:16:16.084390 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:16.084219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:58579)
0621 05:16:16.084324 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:16.084356 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260621 05:16:17.083400 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:17.083215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:49499)
0621 05:16:17.083321 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:17.083353 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
I20260621 05:16:17.083400 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:17.082965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:47023)
0621 05:16:17.083303 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:17.083346 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":61,"threads_started":1}
I20260621 05:16:17.084898 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:17.084772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:43805)
0621 05:16:17.084847 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:17.084868 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64,"thread_start_us":36,"threads_started":1}
I20260621 05:16:18.084169 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:18.083887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:35731)
0621 05:16:18.084074 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:18.084120 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":57,"threads_started":1}
I20260621 05:16:18.084169 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:18.083887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:51409)
0621 05:16:18.084112 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:18.084139 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":59,"threads_started":1}
I20260621 05:16:18.085471 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:18.085316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:49637)
0621 05:16:18.085407 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:18.085438 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":44,"threads_started":1}
I20260621 05:16:18.199981 7836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4adbac03f52493a9fced4f80ec801ea)
I20260621 05:16:18.200064 7836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:18.200242 7836 leader_election.cc:288] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707)
I20260621 05:16:18.200599 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:18.200374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:38204)
0621 05:16:18.200517 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:18.200563 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
W20260621 05:16:18.200680 7064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260621 05:16:18.201012 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:18.200924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:38214)
0621 05:16:18.200971 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:18.200996 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:16:18.201097 7064 leader_election.cc:334] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:18.201146 7064 leader_election.cc:302] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 75a81ca07d054b8b99d42bf8bd453a57; no voters: d4adbac03f52493a9fced4f80ec801ea
I20260621 05:16:18.201247 7836 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:16:19.084780 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:19.084563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:35647)
0621 05:16:19.084658 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:19.084726 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20260621 05:16:19.084790 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:19.084583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:49183)
0621 05:16:19.084688 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:19.084738 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
I20260621 05:16:19.085944 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:19.085795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:36205)
0621 05:16:19.085883 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:19.085912 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":41,"threads_started":1}
I20260621 05:16:20.085459 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:20.085254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:37533)
0621 05:16:20.085364 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:20.085407 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260621 05:16:20.085561 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:20.085155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:35745)
0621 05:16:20.085379 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:20.085512 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":58,"threads_started":1}
I20260621 05:16:20.086457 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:20.086330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:41361)
0621 05:16:20.086406 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:20.086427 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":37,"threads_started":1}
I20260621 05:16:21.086266 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:21.086055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:38389)
0621 05:16:21.086171 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:21.086210 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260621 05:16:21.086441 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:21.086233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:33065)
0621 05:16:21.086336 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:21.086377 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260621 05:16:21.086970 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:21.086823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:35295)
0621 05:16:21.086910 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:21.086939 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":44,"threads_started":1}
I20260621 05:16:22.087260 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:22.087026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:44819)
0621 05:16:22.087158 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:22.087206 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260621 05:16:22.087272 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:22.087026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:41021)
0621 05:16:22.087158 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:22.087238 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260621 05:16:22.087565 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:22.087344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:60383)
0621 05:16:22.087490 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:22.087529 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260621 05:16:23.087941 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:23.087731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:59145)
0621 05:16:23.087847 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:23.087885 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260621 05:16:23.088006 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:23.087686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:40489)
0621 05:16:23.087933 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:23.087964 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":60,"threads_started":1}
I20260621 05:16:23.088181 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:23.087953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:36835)
0621 05:16:23.088088 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:23.088128 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260621 05:16:24.088709 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:24.088501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:59699)
0621 05:16:24.088620 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:24.088656 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260621 05:16:24.088709 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:24.088501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:47441)
0621 05:16:24.088648 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:24.088672 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260621 05:16:24.088734 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:24.088516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:56187)
0621 05:16:24.088656 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:24.088696 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":53,"threads_started":1}
I20260621 05:16:25.089267 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:25.089045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:50925)
0621 05:16:25.089168 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:25.089203 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260621 05:16:25.089257 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:25.089046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:49087)
0621 05:16:25.089168 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:25.089203 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260621 05:16:25.089534 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:25.089362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:39945)
0621 05:16:25.089461 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:25.089497 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
I20260621 05:16:26.089857 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:26.089627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:33423)
0621 05:16:26.089762 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:26.089798 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260621 05:16:26.089988 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:26.089627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:40593)
0621 05:16:26.089916 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:26.089948 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":58,"threads_started":1}
I20260621 05:16:26.090070 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:26.089877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:45649)
0621 05:16:26.089981 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:26.090016 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
I20260621 05:16:27.090641 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:27.090345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:53453)
0621 05:16:27.090475 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:27.090584 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260621 05:16:27.090652 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:27.090463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:44897)
0621 05:16:27.090590 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:27.090616 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":42,"threads_started":1}
I20260621 05:16:27.090641 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:27.090345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:36143)
0621 05:16:27.090519 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:27.090585 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
I20260621 05:16:28.091212 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:28.091006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:49825)
0621 05:16:28.091132 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:28.091163 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":54,"threads_started":1}
I20260621 05:16:28.091281 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:28.091043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:54525)
0621 05:16:28.091207 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:28.091239 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":43,"threads_started":1}
I20260621 05:16:28.091187 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:28.090976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:37115)
0621 05:16:28.091102 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:28.091134 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260621 05:16:29.091846 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:29.091648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:60477)
0621 05:16:29.091768 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:29.091805 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":51,"threads_started":1}
I20260621 05:16:29.091846 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:29.091648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:37423)
0621 05:16:29.091774 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:29.091804 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
I20260621 05:16:29.091826 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:29.091601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:56631)
0621 05:16:29.091732 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:29.091768 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260621 05:16:30.092571 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:30.092338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:56559)
0621 05:16:30.092473 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:30.092510 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260621 05:16:30.092618 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:30.092352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:46783)
0621 05:16:30.092505 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:30.092552 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":55,"threads_started":1}
I20260621 05:16:30.092710 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:30.092505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:37539)
0621 05:16:30.092631 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:30.092665 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260621 05:16:31.093283 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:31.093038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:39335)
0621 05:16:31.093197 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:31.093240 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260621 05:16:31.093309 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:31.093120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:47929)
0621 05:16:31.093226 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:31.093259 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260621 05:16:31.093300 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:31.093140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:33467)
0621 05:16:31.093226 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:31.093258 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":38,"threads_started":1}
I20260621 05:16:31.230115 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4adbac03f52493a9fced4f80ec801ea)
I20260621 05:16:31.230206 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4adbac03f52493a9fced4f80ec801ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35707 } } peers { permanent_uuid: "75a81ca07d054b8b99d42bf8bd453a57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attrs { promote: false } } peers { permanent_uuid: "d2264b1c16f34807ae14238f6be1570e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40775 } attrs { promote: true } }
I20260621 05:16:31.230412 7878 leader_election.cc:288] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707)
I20260621 05:16:31.230706 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:31.230544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:58170)
0621 05:16:31.230635 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:31.230672 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":43,"threads_started":1}
W20260621 05:16:31.230819 7064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:31.231176 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:31.231080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.0.1:58184)
0621 05:16:31.231137 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:31.231161 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:16:31.231276 7064 leader_election.cc:334] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4adbac03f52493a9fced4f80ec801ea (127.0.98.62:35707): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
I20260621 05:16:31.231328 7064 leader_election.cc:302] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 75a81ca07d054b8b99d42bf8bd453a57; no voters: d4adbac03f52493a9fced4f80ec801ea
I20260621 05:16:31.231415 7878 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75a81ca07d054b8b99d42bf8bd453a57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:16:32.093880 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:32.093659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:53919)
0621 05:16:32.093789 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:32.093825 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260621 05:16:32.093879 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:32.093705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:55449)
0621 05:16:32.093799 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:32.093839 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
I20260621 05:16:32.093879 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:32.093660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:47857)
0621 05:16:32.093809 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:32.093839 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260621 05:16:33.094612 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:33.094331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.2:45831)
0621 05:16:33.094514 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:33.094553 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":52,"threads_started":1}
I20260621 05:16:33.094800 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:33.094317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.1:56805)
0621 05:16:33.094625 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:33.094656 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":55,"threads_started":1}
I20260621 05:16:33.094787 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:33.094478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35707 (local address 127.0.98.3:55349)
0621 05:16:33.094637 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:33.094670 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":37,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:35707,127.0.98.61:45645,127.0.98.60:40775, num_attempts: 1)
I20260621 05:16:33.248816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 7226
I20260621 05:16:33.254370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 7452
I20260621 05:16:33.258834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 7588
I20260621 05:16:33.262894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 6945
I20260621 05:16:33.266880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 7132
2026-06-21T05:16:33Z chronyd exiting
I20260621 05:16:33.282671 392 test_util.cc:175] -----------------------------------------------
I20260621 05:16:33.282711 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018933478224-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32931 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:16:33 dist-test-slave-czm5 krb5kdc[7889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:16:33 dist-test-slave-czm5 krb5kdc[7889](info): set up 2 sockets
Jun 21 05:16:33 dist-test-slave-czm5 krb5kdc[7889](info): commencing operation
krb5kdc: starting...
W20260621 05:16:34.252053 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:16:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260621 05:16:34.356158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:34.433519 7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:34.433686 7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:34.433705 7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:34.434758 7902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:34.434796 7902 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:16:34.434808 7902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:34.434821 7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:34.434834 7902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:34.434844 7902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:34.436213 7902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37085
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:34.436390 7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:34.436532 7902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:34.441339 7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:34.441362 7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:34.441380 7902 server_base.cc:1034] running on GCE node
W20260621 05:16:34.441363 7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:34.441802 7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:34.442108 7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:34.443300 7902 hybrid_clock.cc:648] HybridClock initialized: now 1782018994443290 us; error 27 us; skew 500 ppm
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:34.445636 7902 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260621 05:16:34.446031 7902 webserver.cc:460] Webserver started at http://127.0.98.62:34903/ using document root <none> and password file <none>
I20260621 05:16:34.446256 7902 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:34.446327 7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:34.446487 7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:34.447436 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "23e3437c9b31449fae200708d2356446"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "3a58c8ea6435f3feeeddcbfbf0695821"
server_key_iv: "7d8d3bc792d5eba8298ad4952522fe8b"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.447803 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "23e3437c9b31449fae200708d2356446"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "3a58c8ea6435f3feeeddcbfbf0695821"
server_key_iv: "7d8d3bc792d5eba8298ad4952522fe8b"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.449164 7902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:34.449816 7917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:34.449995 7902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:34.450083 7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "23e3437c9b31449fae200708d2356446"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "3a58c8ea6435f3feeeddcbfbf0695821"
server_key_iv: "7d8d3bc792d5eba8298ad4952522fe8b"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.450131 7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:34.457465 7902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:34.459079 7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:34.459175 7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:34.463721 7902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37085
I20260621 05:16:34.463781 7979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37085 every 8 connection(s)
I20260621 05:16:34.464056 7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:16:34.464614 7980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:34.466615 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: Bootstrap starting.
I20260621 05:16:34.467155 7980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:34.467422 7980 log.cc:824] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:34.468055 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: No bootstrap required, opened a new log
I20260621 05:16:34.469956 7980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3437c9b31449fae200708d2356446" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37085 } }
I20260621 05:16:34.470173 7980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:34.470216 7980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e3437c9b31449fae200708d2356446, State: Initialized, Role: FOLLOWER
I20260621 05:16:34.470224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 7902
I20260621 05:16:34.470317 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:16:34.470366 7980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3437c9b31449fae200708d2356446" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37085 } }
I20260621 05:16:34.470449 7980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:16:34.470489 7980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:16:34.470530 392 external_mini_cluster.cc:1428] Setting key 1072e2c04e1fd9d4c4f7e1d1da43720b
I20260621 05:16:34.470537 7980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:34.471235 7980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3437c9b31449fae200708d2356446" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37085 } }
I20260621 05:16:34.471314 7980 leader_election.cc:302] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23e3437c9b31449fae200708d2356446; no voters:
I20260621 05:16:34.471477 7980 leader_election.cc:288] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:16:34.471539 7985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:34.471728 7985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [term 1 LEADER]: Becoming Leader. State: Replica: 23e3437c9b31449fae200708d2356446, State: Running, Role: LEADER
I20260621 05:16:34.471841 7980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:34.471861 7985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3437c9b31449fae200708d2356446" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37085 } }
I20260621 05:16:34.472206 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e3437c9b31449fae200708d2356446. Latest consensus state: current_term: 1 leader_uuid: "23e3437c9b31449fae200708d2356446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3437c9b31449fae200708d2356446" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37085 } } }
I20260621 05:16:34.472430 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:34.472354 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23e3437c9b31449fae200708d2356446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3437c9b31449fae200708d2356446" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37085 } } }
I20260621 05:16:34.472772 7994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:16:34.472779 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446 [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:34.473264 7994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:16:34.474615 7994 catalog_manager.cc:1347] Generated new cluster ID: c996521aef2b4c7480eae3f3df4fa416
I20260621 05:16:34.474664 7994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:34.481489 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.470967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.0.98.62:37085)
0621 05:16:34.471135 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.471168 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.471202 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:16:34.471296 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.471302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.471358 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.471512 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.472296 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.473620 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.474647 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.475255 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.475331 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.478062 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:34.478066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:34.478066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:34.478076 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:34.479962 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.480359 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.480362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.480362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.480462 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.480674 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.480674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.480675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.480934 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:34.481082 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.481201 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.481316 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":102,"threads_started":1}
I20260621 05:16:34.489500 7994 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:16:34.490005 7994 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:16:34.495260 7994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: Generated new TSK 0
I20260621 05:16:34.495427 7994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260621 05:16:34.521998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37085
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:34.598980 8006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:34.599200 8006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:34.599253 8006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:34.600490 8006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:34.600556 8006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:34.600593 8006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:16:34.600649 8006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:16:34.602049 8006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:34.602288 8006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:34.602504 8006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:34.607034 8011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:34.607035 8014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:34.607108 8012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:34.607127 8006 server_base.cc:1034] running on GCE node
I20260621 05:16:34.607370 8006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:34.607666 8006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:34.608816 8006 hybrid_clock.cc:648] HybridClock initialized: now 1782018994608804 us; error 29 us; skew 500 ppm
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:34.611097 8006 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260621 05:16:34.611558 8006 webserver.cc:460] Webserver started at http://127.0.98.1:38887/ using document root <none> and password file <none>
I20260621 05:16:34.611837 8006 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:34.611909 8006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:34.612052 8006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:34.613023 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "6e1223829ee24eaa8930b3a815a3daac"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "00d6dfcf5e2ec21b6dca58f12e0f9351"
server_key_iv: "9f024494f4e1d36569126dfb0b512577"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.613372 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6e1223829ee24eaa8930b3a815a3daac"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "00d6dfcf5e2ec21b6dca58f12e0f9351"
server_key_iv: "9f024494f4e1d36569126dfb0b512577"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.614627 8006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:34.615389 8021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:34.615561 8006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:34.615672 8006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "6e1223829ee24eaa8930b3a815a3daac"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "00d6dfcf5e2ec21b6dca58f12e0f9351"
server_key_iv: "9f024494f4e1d36569126dfb0b512577"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.615756 8006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:34.623613 8006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:34.625345 8006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:34.625494 8006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:34.625980 8006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:34.626036 8006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:34.626075 8006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:34.626102 8006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:34.632926 8006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43327
I20260621 05:16:34.633002 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43327 every 8 connection(s)
I20260621 05:16:34.633229 8006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:16:34.634327 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.626351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54109 (local address 127.0.98.62:37085)
0621 05:16:34.626415 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.626418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.626596 (+ 178us) server_negotiation.cc:407] Connection header received
0621 05:16:34.626976 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.626978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.626994 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.627076 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.627585 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.628104 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.629094 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.629295 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.629374 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.631757 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:34.631758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:34.631759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:34.631769 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:34.633284 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.633718 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.633718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.633719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.633765 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.633935 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.633936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.633936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.634019 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:34.634065 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.634221 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.634273 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:16:34.634476 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.626343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37085 (local address 127.0.98.1:54109)
0621 05:16:34.626474 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.626509 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.626891 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.627118 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.627120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.627322 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:34.627491 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.627494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.628191 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.628191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.628996 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.628999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.629384 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.629385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.629494 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.629938 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:34.629945 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:34.631600 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:34.633366 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.633370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.633372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.633645 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.633808 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.633808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.633808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.633885 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.634059 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:34.634061 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:34.634147 (+ 86us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.634210 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.634304 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260621 05:16:34.634970 8137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37085
I20260621 05:16:34.635049 8137 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:34.635226 8137 heartbeater.cc:511] Master 127.0.98.62:37085 requested a full tablet report, sending...
I20260621 05:16:34.635773 7934 ts_manager.cc:194] Registered new tserver with Master: 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1:43327)
I20260621 05:16:34.636200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 8006
I20260621 05:16:34.636296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:16:34.636461 392 external_mini_cluster.cc:1428] Setting key 2afcf5e57404e83147e072db0425b97b
I20260621 05:16:34.636657 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54109
I20260621 05:16:34.642423 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.635977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37085 (local address 127.0.98.1:59631)
0621 05:16:34.636088 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.636097 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.636147 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.636978 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.636979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.637157 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:34.637228 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.637229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.637891 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.637892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.638709 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.638711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.639076 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.639077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.639142 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.639518 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:34.639522 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:34.639832 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:34.641564 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.641565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.641565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.641728 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.641923 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.641923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.641923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.641969 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.642225 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:34.642225 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:34.642254 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.642298 (+ 44us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.642364 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":614,"threads_started":1}
I20260621 05:16:34.642516 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.636034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59631 (local address 127.0.98.62:37085)
0621 05:16:34.636845 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.636847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.636856 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:34.636872 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.636872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.636880 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.636940 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.637309 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.637800 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.638799 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.638995 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.639073 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.639908 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:34.639909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:34.639910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:34.639916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:34.641453 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.641795 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.641796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.641796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.641854 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.642024 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.642025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.642025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.642162 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:34.642232 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.642324 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.642394 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260621 05:16:34.661888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37085
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:34.737388 8146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:34.737567 8146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:34.737591 8146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:34.738692 8146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:34.738734 8146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:34.738754 8146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:16:34.738813 8146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:34.740309 8146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:34.740533 8146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:34.740751 8146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:16:34.745446 8146 server_base.cc:1034] running on GCE node
W20260621 05:16:34.745466 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:34.745379 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:34.745379 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:34.745764 8146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:34.746001 8146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:34.747165 8146 hybrid_clock.cc:648] HybridClock initialized: now 1782018994747144 us; error 47 us; skew 500 ppm
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:34.749433 8146 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260621 05:16:34.749876 8146 webserver.cc:460] Webserver started at http://127.0.98.2:41851/ using document root <none> and password file <none>
I20260621 05:16:34.750113 8146 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:34.750177 8146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:34.750306 8146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:34.751281 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "1a8e095a154d4405a9468e5513e1140a"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "f907e5e3c355ad4f3ab8c10c4b72e8e2"
server_key_iv: "7995a79e58c5ec09d241be2a8deacef2"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.751677 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1a8e095a154d4405a9468e5513e1140a"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "f907e5e3c355ad4f3ab8c10c4b72e8e2"
server_key_iv: "7995a79e58c5ec09d241be2a8deacef2"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.753006 8146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:34.753718 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:34.753995 8146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:34.754083 8146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "1a8e095a154d4405a9468e5513e1140a"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "f907e5e3c355ad4f3ab8c10c4b72e8e2"
server_key_iv: "7995a79e58c5ec09d241be2a8deacef2"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.754159 8146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:34.769429 8146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:34.771047 8146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:34.771196 8146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:34.771801 8146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:34.771858 8146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:34.771899 8146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:34.771936 8146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:34.778824 8146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33121
I20260621 05:16:34.779139 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33121 every 8 connection(s)
I20260621 05:16:34.779217 8146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:16:34.781121 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.772217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39455 (local address 127.0.98.62:37085)
0621 05:16:34.772255 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.772257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.772504 (+ 247us) server_negotiation.cc:407] Connection header received
0621 05:16:34.773017 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.773018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.773030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.773089 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.773722 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.774237 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.775750 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.776046 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.776121 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.778590 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:34.778592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:34.778593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:34.778601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:34.780084 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.780419 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.780419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.780420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.780464 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.780657 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.780657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.780658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.780749 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:34.780795 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.781006 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.781061 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:16:34.781267 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.772196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37085 (local address 127.0.98.2:39455)
0621 05:16:34.772328 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.772411 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.772918 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.773149 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.773153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.773374 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:34.773629 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.773633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.774326 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.774327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.775634 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.775637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.776145 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.776146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.776276 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.776811 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:34.776819 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:34.778486 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:34.780152 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.780156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.780157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.780354 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.780520 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.780520 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.780520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.780605 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.780800 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:34.780803 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:34.780900 (+ 97us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.780978 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.781083 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260621 05:16:34.781689 8277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37085
I20260621 05:16:34.781764 8277 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:34.781926 8277 heartbeater.cc:511] Master 127.0.98.62:37085 requested a full tablet report, sending...
I20260621 05:16:34.782325 7934 ts_manager.cc:194] Registered new tserver with Master: 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121)
I20260621 05:16:34.782840 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39455
I20260621 05:16:34.786118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 8146
I20260621 05:16:34.786222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:16:34.786406 392 external_mini_cluster.cc:1428] Setting key d32dcfc9e97f87651092eb266158c2c8
I20260621 05:16:34.787545 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.782552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37085 (local address 127.0.98.2:55861)
0621 05:16:34.782696 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.782705 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.782759 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.782880 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.782881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.783086 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:34.783140 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.783142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.783914 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.783914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.784385 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.784386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.784674 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.784675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.784716 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.785095 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:34.785105 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:34.785329 (+ 224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:34.786612 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.786613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.786614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.786797 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.786975 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.786976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.786976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.787037 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.787338 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:34.787339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:34.787369 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.787417 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.787483 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260621 05:16:34.787663 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.782635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55861 (local address 127.0.98.62:37085)
0621 05:16:34.782671 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.782672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.782762 (+ 90us) server_negotiation.cc:407] Connection header received
0621 05:16:34.782809 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.782809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.782818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.782879 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.783191 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.783837 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.784449 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.784606 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.784662 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.785390 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:34.785391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:34.785391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:34.785395 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:34.786541 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.786862 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.786863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.786863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.786917 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.787099 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.787100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.787100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.787274 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:34.787339 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.787443 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.787516 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260621 05:16:34.808403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37085
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:34.885506 8286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:34.885670 8286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:34.885691 8286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:34.886684 8286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:34.886726 8286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:34.886744 8286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:16:34.886759 8286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:34.888087 8286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:34.888249 8286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:34.888442 8286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:34.893010 8291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:34.893020 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:34.893020 8292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:34.893293 8286 server_base.cc:1034] running on GCE node
I20260621 05:16:34.893481 8286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:34.893700 8286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:34.894856 8286 hybrid_clock.cc:648] HybridClock initialized: now 1782018994894836 us; error 32 us; skew 500 ppm
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:34.897130 8286 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260621 05:16:34.897452 8286 webserver.cc:460] Webserver started at http://127.0.98.3:40973/ using document root <none> and password file <none>
I20260621 05:16:34.897603 8286 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:34.897639 8286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:34.897713 8286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:34.898576 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "67db0b30d25d480dae0e979b4cf71a6b"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "c1749a9f490b74d9cf81cd07e75a8426"
server_key_iv: "a1bd7ef584c73af93998858d7eefdf83"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.898839 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "67db0b30d25d480dae0e979b4cf71a6b"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "c1749a9f490b74d9cf81cd07e75a8426"
server_key_iv: "a1bd7ef584c73af93998858d7eefdf83"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.900048 8286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:16:34.900676 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:34.900830 8286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:34.900905 8286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "67db0b30d25d480dae0e979b4cf71a6b"
format_stamp: "Formatted at 2026-06-21 05:16:34 on dist-test-slave-czm5"
server_key: "c1749a9f490b74d9cf81cd07e75a8426"
server_key_iv: "a1bd7ef584c73af93998858d7eefdf83"
server_key_version: "encryptionkey@0"
I20260621 05:16:34.900954 8286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:34.919763 8286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:34.921828 8286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:34.921941 8286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:34.922374 8286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:34.922415 8286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:34.922433 8286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:34.922467 8286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:34.929008 8286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36547
I20260621 05:16:34.929128 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36547 every 8 connection(s)
I20260621 05:16:34.929358 8286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:16:34.932003 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.922812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57625 (local address 127.0.98.62:37085)
0621 05:16:34.922876 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.922878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.923055 (+ 177us) server_negotiation.cc:407] Connection header received
0621 05:16:34.923448 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.923449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.923462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.923536 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.924228 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.924972 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.926475 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.926813 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.926893 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.929479 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:34.929480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:34.929481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:34.929489 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:34.930894 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.931233 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.931233 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.931234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.931277 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.931482 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.931482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.931482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.931569 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:34.931614 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.931891 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.931946 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:16:34.932176 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.922802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37085 (local address 127.0.98.3:57625)
0621 05:16:34.922948 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.922981 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.923370 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.923579 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.923581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.923881 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:34.924133 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.924138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.925063 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.925064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.926391 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.926394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.926914 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.926915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.927066 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.927521 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:34.927532 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:34.929367 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:34.930960 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.930963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.930964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.931172 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.931345 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.931345 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.931346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.931433 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.931656 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:34.931661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:34.931786 (+ 125us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.931864 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.931969 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":68,"threads_started":1}
I20260621 05:16:34.932411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 8286
I20260621 05:16:34.932531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:16:34.932736 8417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37085
I20260621 05:16:34.932766 392 external_mini_cluster.cc:1428] Setting key eb5eb0b563215ef3e5abe72dcd70ae0c
I20260621 05:16:34.932832 8417 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:34.932991 8417 heartbeater.cc:511] Master 127.0.98.62:37085 requested a full tablet report, sending...
I20260621 05:16:34.933346 7933 ts_manager.cc:194] Registered new tserver with Master: 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:34.933794 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57625
I20260621 05:16:34.938373 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.933679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40239 (local address 127.0.98.62:37085)
0621 05:16:34.933714 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.933715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.933771 (+ 56us) server_negotiation.cc:407] Connection header received
0621 05:16:34.933838 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.933839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.933847 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.933894 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.934213 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.934771 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.935473 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.935737 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.935806 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.936610 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:34.936610 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:34.936610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:34.936615 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:34.937647 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.937930 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.937930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.937930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.937971 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.938072 (+ 101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.938072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.938072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.938163 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:34.938215 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.938285 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.938327 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:16:34.938390 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.933622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37085 (local address 127.0.98.3:40239)
0621 05:16:34.933726 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.933735 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.933786 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.933908 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.933908 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.934071 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:34.934137 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.934138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.934852 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.934853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.935411 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.935412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.935831 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.935832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.935882 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.936304 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:34.936308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:34.936547 (+ 239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:34.937710 (+ 1163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.937711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.937711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.937858 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.938006 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:34.938006 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:34.938006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:34.938043 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:34.938195 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:34.938195 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:34.938232 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.938273 (+ 41us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.938334 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260621 05:16:34.944073 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:16:34.963279 7984 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:34.958576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.0.98.62:37085)
0621 05:16:34.958617 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.958618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.958638 (+ 20us) server_negotiation.cc:407] Connection header received
0621 05:16:34.958687 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.958687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.958694 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.958737 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.959253 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.959782 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.960465 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.960633 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.960683 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.963206 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50764: BlockingRecv error: recv got EOF from 127.0.0.1:50764 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 21 05:16:34 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:34.969678 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.964016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.0.98.62:37085)
0621 05:16:34.964055 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.964057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.964125 (+ 68us) server_negotiation.cc:407] Connection header received
0621 05:16:34.964161 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.964161 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.964168 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.964232 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:34.964634 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.965055 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.965659 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.965819 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.965864 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.967651 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:34.967654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:34.967654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:34.967675 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:34.968978 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.969235 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.969236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.969236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.969271 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:34.969392 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:34.969392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:34.969392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:34.969462 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:34.969518 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.969590 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.969632 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:16:34.971179 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:16:34.972877 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:16:34.978806 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.974681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33121 (local address 127.0.0.1:46064)
0621 05:16:34.975035 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.975045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.975097 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.975354 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.975355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.975357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:34.975407 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.975409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.976398 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.976399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.977718 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.977720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.978476 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.978477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.978561 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.978594 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.978687 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.978730 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":50,"threads_started":1}
I20260621 05:16:34.978825 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.974831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.0.98.2:33121)
0621 05:16:34.975207 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.975211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.975219 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:34.975258 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.975259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.975270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.975356 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:34.975554 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.976318 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.977795 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.978388 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.978460 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.978539 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.978698 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.978733 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":91,"threads_started":1}
I20260621 05:16:34.979269 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.974877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36547 (local address 127.0.0.1:45318)
0621 05:16:34.975423 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.975432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.975482 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.975648 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.975649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.975650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:34.975681 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.975682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.976845 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.976845 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.978057 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.978059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.979009 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.979010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.979080 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.979085 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.979139 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.979187 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":34,"thread_start_us":42,"threads_started":1}
I20260621 05:16:34.979323 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.974652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.0.1:48724)
0621 05:16:34.974837 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:34.974872 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:34.974935 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:34.975254 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:34.975257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:34.975259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:34.975310 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.975311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.976844 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.976845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.978057 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:34.978059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.979039 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.979040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:34.979110 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.979114 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:16:34.979169 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 05:16:34.979208 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":51,"thread_start_us":73,"threads_started":1}
I20260621 05:16:34.979311 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.974756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.0.98.1:43327)
0621 05:16:34.975030 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.975035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.975043 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:34.975082 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.975083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.975095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.975183 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:34.975392 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.976760 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.978153 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.978859 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.978930 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.979007 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.979185 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.979217 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
I20260621 05:16:34.979851 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:34.975036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.0.98.3:36547)
0621 05:16:34.975190 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:34.975194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:34.975495 (+ 301us) server_negotiation.cc:407] Connection header received
0621 05:16:34.975544 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:34.975546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:34.975561 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:34.975652 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:34.975737 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.976728 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.978141 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:34.978940 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:34.979008 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:34.979075 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:34.979708 (+ 633us) server_negotiation.cc:299] Negotiation successful
0621 05:16:34.979753 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260621 05:16:34.979734 8211 tablet_service.cc:1467] Processing CreateTablet for tablet addab55a3b1247b594dc7bd85a8dd7bb (DEFAULT_TABLE table=test-table [id=1135d8e088d340349dbf5d82d8fb4432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:34.980106 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addab55a3b1247b594dc7bd85a8dd7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:34.980309 8071 tablet_service.cc:1467] Processing CreateTablet for tablet addab55a3b1247b594dc7bd85a8dd7bb (DEFAULT_TABLE table=test-table [id=1135d8e088d340349dbf5d82d8fb4432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:34.980516 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addab55a3b1247b594dc7bd85a8dd7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:34.980415 8351 tablet_service.cc:1467] Processing CreateTablet for tablet addab55a3b1247b594dc7bd85a8dd7bb (DEFAULT_TABLE table=test-table [id=1135d8e088d340349dbf5d82d8fb4432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:34.980710 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addab55a3b1247b594dc7bd85a8dd7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:34.983109 8442 tablet_bootstrap.cc:492] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: Bootstrap starting.
I20260621 05:16:34.983877 8442 tablet_bootstrap.cc:654] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:34.983872 8443 tablet_bootstrap.cc:492] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Bootstrap starting.
I20260621 05:16:34.984217 8442 log.cc:824] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:34.984167 8444 tablet_bootstrap.cc:492] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: Bootstrap starting.
I20260621 05:16:34.984674 8443 tablet_bootstrap.cc:654] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:34.984818 8444 tablet_bootstrap.cc:654] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:34.984993 8442 tablet_bootstrap.cc:492] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: No bootstrap required, opened a new log
I20260621 05:16:34.985013 8443 log.cc:824] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:34.985066 8442 ts_tablet_manager.cc:1397] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:16:34.985145 8444 log.cc:824] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:34.985783 8443 tablet_bootstrap.cc:492] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: No bootstrap required, opened a new log
I20260621 05:16:34.985818 8444 tablet_bootstrap.cc:492] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: No bootstrap required, opened a new log
I20260621 05:16:34.985846 8443 ts_tablet_manager.cc:1397] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:16:34.985898 8444 ts_tablet_manager.cc:1397] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:16:34.987017 8442 raft_consensus.cc:348] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:34.987181 8442 raft_consensus.cc:374] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:34.987215 8442 raft_consensus.cc:729] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Initialized, Role: FOLLOWER
I20260621 05:16:34.987121 8444 raft_consensus.cc:348] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:34.987263 8444 raft_consensus.cc:374] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:34.987296 8444 raft_consensus.cc:729] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67db0b30d25d480dae0e979b4cf71a6b, State: Initialized, Role: FOLLOWER
I20260621 05:16:34.987313 8442 consensus_queue.cc:260] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:34.987417 8444 consensus_queue.cc:260] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:34.987571 8442 ts_tablet_manager.cc:1428] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 05:16:34.987737 8137 heartbeater.cc:503] Master 127.0.98.62:37085 was elected leader, sending a full tablet report...
I20260621 05:16:34.987735 8443 raft_consensus.cc:348] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:34.987872 8443 raft_consensus.cc:374] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:34.987910 8443 raft_consensus.cc:729] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8e095a154d4405a9468e5513e1140a, State: Initialized, Role: FOLLOWER
I20260621 05:16:34.987663 8444 ts_tablet_manager.cc:1428] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:16:34.988022 8443 consensus_queue.cc:260] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:34.988272 8443 ts_tablet_manager.cc:1428] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:16:34.988284 8277 heartbeater.cc:503] Master 127.0.98.62:37085 was elected leader, sending a full tablet report...
I20260621 05:16:34.988397 8417 heartbeater.cc:503] Master 127.0.98.62:37085 was elected leader, sending a full tablet report...
W20260621 05:16:35.029711 8278 tablet.cc:2369] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:16:35.072695 8448 raft_consensus.cc:482] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:16:35.072790 8448 raft_consensus.cc:504] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.073138 8448 leader_election.cc:288] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.076810 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.073394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51911 (local address 127.0.98.2:33121)
0621 05:16:35.073453 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.073455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.073468 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:16:35.073498 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.073500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.073512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.073575 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:35.073750 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.074499 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.075919 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.076459 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.076542 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.076551 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.076709 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.076745 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:16:35.076810 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.073544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56855 (local address 127.0.98.3:36547)
0621 05:16:35.073608 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.073610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.073744 (+ 134us) server_negotiation.cc:407] Connection header received
0621 05:16:35.073798 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.073799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.073809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.073874 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:35.073993 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.074597 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.075918 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.076457 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.076542 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.076551 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.076709 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.076745 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:16:35.076843 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.073464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36547 (local address 127.0.98.1:56855)
0621 05:16:35.073678 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.073687 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.073739 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.073877 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.073877 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.073878 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:35.073923 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.073931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.074664 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.074664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.075833 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.075836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.076550 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.076550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.076628 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.076632 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.076688 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.076728 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":4,"thread_start_us":104,"threads_started":1}
I20260621 05:16:35.076849 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.073355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33121 (local address 127.0.98.1:51911)
0621 05:16:35.073393 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.073404 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.073446 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.073582 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.073583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.073585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:35.073653 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.073656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.074571 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.074571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.075838 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.075840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.076549 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.076550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.076635 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.076637 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.076688 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.076728 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":54}
I20260621 05:16:35.077412 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addab55a3b1247b594dc7bd85a8dd7bb" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b" is_pre_election: true
I20260621 05:16:35.077399 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addab55a3b1247b594dc7bd85a8dd7bb" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8e095a154d4405a9468e5513e1140a" is_pre_election: true
I20260621 05:16:35.077545 8231 raft_consensus.cc:2454] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 0.
I20260621 05:16:35.077545 8371 raft_consensus.cc:2454] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 0.
I20260621 05:16:35.077744 8024 leader_election.cc:302] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67db0b30d25d480dae0e979b4cf71a6b, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.077853 8448 raft_consensus.cc:2781] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:16:35.077903 8448 raft_consensus.cc:482] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:16:35.077926 8448 raft_consensus.cc:3037] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.078773 8448 raft_consensus.cc:504] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.078890 8448 leader_election.cc:288] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 election: Requested vote from peers 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.079051 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addab55a3b1247b594dc7bd85a8dd7bb" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b"
I20260621 05:16:35.079051 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addab55a3b1247b594dc7bd85a8dd7bb" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8e095a154d4405a9468e5513e1140a"
I20260621 05:16:35.079126 8371 raft_consensus.cc:3037] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.079141 8231 raft_consensus.cc:3037] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.079842 8371 raft_consensus.cc:2454] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 1.
I20260621 05:16:35.079909 8231 raft_consensus.cc:2454] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 1.
I20260621 05:16:35.080016 8024 leader_election.cc:302] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67db0b30d25d480dae0e979b4cf71a6b, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.080132 8448 raft_consensus.cc:2781] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:35.080277 8448 raft_consensus.cc:686] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 1 LEADER]: Becoming Leader. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Running, Role: LEADER
I20260621 05:16:35.080346 8448 consensus_queue.cc:237] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.081161 7934 catalog_manager.cc:5495] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } health_report { overall_health: UNKNOWN } } }
I20260621 05:16:35.094367 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.092063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.0.98.1:43327)
0621 05:16:35.092121 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.092124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.092132 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:35.092201 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.092202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.092219 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.092284 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:35.092405 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.092843 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.093604 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.093809 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.093884 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.094046 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.094145 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.094227 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.094276 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.094294 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:16:35.096015 8231 raft_consensus.cc:1261] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Refusing update from remote peer 6e1223829ee24eaa8930b3a815a3daac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.096035 8371 raft_consensus.cc:1261] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Refusing update from remote peer 6e1223829ee24eaa8930b3a815a3daac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.096392 8448 consensus_queue.cc:1035] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.096514 8452 consensus_queue.cc:1035] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Connected to new peer: Peer: permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.100886 7933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:35.100970 7933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:35.101747 7933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: Sending DeleteTablet for 3 replicas of tablet addab55a3b1247b594dc7bd85a8dd7bb
I20260621 05:16:35.102046 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet addab55a3b1247b594dc7bd85a8dd7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46064
I20260621 05:16:35.102057 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet addab55a3b1247b594dc7bd85a8dd7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48724
I20260621 05:16:35.102082 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet addab55a3b1247b594dc7bd85a8dd7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45318
I20260621 05:16:35.102363 8465 tablet_replica.cc:330] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: stopping tablet replica
I20260621 05:16:35.102509 8465 raft_consensus.cc:2229] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:16:35.102464 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 05:16:35.102566 8464 tablet_replica.cc:330] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: stopping tablet replica
W20260621 05:16:35.102592 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:16:35.102661 8464 raft_consensus.cc:2229] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:35.102753 8464 raft_consensus.cc:2258] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.103143 8464 ts_tablet_manager.cc:1905] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.103734 8211 tablet_service.cc:1467] Processing CreateTablet for tablet d9739e97e517478f8dc8a94dcc593941 (DEFAULT_TABLE table=test-table [id=7a1ae499f10c4ed0973c05648609c403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.103787 8351 tablet_service.cc:1467] Processing CreateTablet for tablet d9739e97e517478f8dc8a94dcc593941 (DEFAULT_TABLE table=test-table [id=7a1ae499f10c4ed0973c05648609c403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.103837 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9739e97e517478f8dc8a94dcc593941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.103868 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9739e97e517478f8dc8a94dcc593941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.103883 8071 tablet_service.cc:1467] Processing CreateTablet for tablet d9739e97e517478f8dc8a94dcc593941 (DEFAULT_TABLE table=test-table [id=7a1ae499f10c4ed0973c05648609c403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.103981 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9739e97e517478f8dc8a94dcc593941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.105070 8443 tablet_bootstrap.cc:492] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: Bootstrap starting.
I20260621 05:16:35.105221 8464 ts_tablet_manager.cc:1918] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:35.105301 8464 log.cc:1196] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/addab55a3b1247b594dc7bd85a8dd7bb
I20260621 05:16:35.105490 8444 tablet_bootstrap.cc:492] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: Bootstrap starting.
I20260621 05:16:35.105499 8443 tablet_bootstrap.cc:654] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.105549 8464 ts_tablet_manager.cc:1939] T addab55a3b1247b594dc7bd85a8dd7bb P 1a8e095a154d4405a9468e5513e1140a: Deleting consensus metadata
I20260621 05:16:35.106139 8444 tablet_bootstrap.cc:654] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.106206 8443 tablet_bootstrap.cc:492] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: No bootstrap required, opened a new log
I20260621 05:16:35.106245 8443 ts_tablet_manager.cc:1397] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:35.106370 8443 raft_consensus.cc:348] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.106417 8442 tablet_bootstrap.cc:492] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: Bootstrap starting.
I20260621 05:16:35.106441 8443 raft_consensus.cc:374] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.106464 8443 raft_consensus.cc:729] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8e095a154d4405a9468e5513e1140a, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.106508 8443 consensus_queue.cc:260] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.106580 8443 ts_tablet_manager.cc:1428] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:35.106663 8444 tablet_bootstrap.cc:492] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: No bootstrap required, opened a new log
I20260621 05:16:35.106700 8444 ts_tablet_manager.cc:1397] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:35.106797 8442 tablet_bootstrap.cc:654] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.106879 8444 raft_consensus.cc:348] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.106947 8444 raft_consensus.cc:374] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.106971 8444 raft_consensus.cc:729] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67db0b30d25d480dae0e979b4cf71a6b, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.107005 8444 consensus_queue.cc:260] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.107103 8444 ts_tablet_manager.cc:1428] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.107321 7920 catalog_manager.cc:4841] TS 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121): tablet addab55a3b1247b594dc7bd85a8dd7bb (table test-table [id=1135d8e088d340349dbf5d82d8fb4432]) successfully deleted
I20260621 05:16:35.107465 8442 tablet_bootstrap.cc:492] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: No bootstrap required, opened a new log
I20260621 05:16:35.107502 8442 ts_tablet_manager.cc:1397] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:35.107663 8442 raft_consensus.cc:348] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.107744 8442 raft_consensus.cc:374] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.107769 8442 raft_consensus.cc:729] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.103181 8465 raft_consensus.cc:2258] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.107808 8442 consensus_queue.cc:260] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.107944 8442 ts_tablet_manager.cc:1428] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:35.108039 8466 tablet_replica.cc:330] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: stopping tablet replica
I20260621 05:16:35.108139 8466 raft_consensus.cc:2229] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:35.108255 8466 raft_consensus.cc:2258] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.108659 8466 ts_tablet_manager.cc:1905] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.108784 8465 ts_tablet_manager.cc:1905] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.109967 8465 ts_tablet_manager.cc:1918] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:35.110054 8465 log.cc:1196] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/addab55a3b1247b594dc7bd85a8dd7bb
I20260621 05:16:35.110169 8466 ts_tablet_manager.cc:1918] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:35.110241 8466 log.cc:1196] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/addab55a3b1247b594dc7bd85a8dd7bb
I20260621 05:16:35.110337 8465 ts_tablet_manager.cc:1939] T addab55a3b1247b594dc7bd85a8dd7bb P 6e1223829ee24eaa8930b3a815a3daac: Deleting consensus metadata
I20260621 05:16:35.110507 8466 ts_tablet_manager.cc:1939] T addab55a3b1247b594dc7bd85a8dd7bb P 67db0b30d25d480dae0e979b4cf71a6b: Deleting consensus metadata
I20260621 05:16:35.110714 7919 catalog_manager.cc:4841] TS 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1:43327): tablet addab55a3b1247b594dc7bd85a8dd7bb (table test-table [id=1135d8e088d340349dbf5d82d8fb4432]) successfully deleted
I20260621 05:16:35.110862 7920 catalog_manager.cc:4841] TS 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547): tablet addab55a3b1247b594dc7bd85a8dd7bb (table test-table [id=1135d8e088d340349dbf5d82d8fb4432]) successfully deleted
W20260621 05:16:35.133873 8138 tablet.cc:2369] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:16:35.179872 8418 tablet.cc:2369] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:16:35.361506 8448 raft_consensus.cc:482] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:16:35.361608 8448 raft_consensus.cc:504] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.361801 8448 leader_election.cc:288] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.362027 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9739e97e517478f8dc8a94dcc593941" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b" is_pre_election: true
I20260621 05:16:35.362109 8371 raft_consensus.cc:2454] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 0.
I20260621 05:16:35.362044 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9739e97e517478f8dc8a94dcc593941" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8e095a154d4405a9468e5513e1140a" is_pre_election: true
I20260621 05:16:35.362157 8231 raft_consensus.cc:2454] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 0.
I20260621 05:16:35.362265 8024 leader_election.cc:302] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67db0b30d25d480dae0e979b4cf71a6b, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.362387 8448 raft_consensus.cc:2781] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:16:35.362447 8448 raft_consensus.cc:482] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:16:35.362488 8448 raft_consensus.cc:3037] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.363174 8448 raft_consensus.cc:504] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.363286 8448 leader_election.cc:288] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 election: Requested vote from peers 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.363485 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9739e97e517478f8dc8a94dcc593941" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b"
I20260621 05:16:35.363494 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9739e97e517478f8dc8a94dcc593941" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8e095a154d4405a9468e5513e1140a"
I20260621 05:16:35.363559 8231 raft_consensus.cc:3037] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.363559 8371 raft_consensus.cc:3037] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.364286 8231 raft_consensus.cc:2454] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 1.
I20260621 05:16:35.364303 8371 raft_consensus.cc:2454] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 1.
I20260621 05:16:35.364442 8024 leader_election.cc:302] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67db0b30d25d480dae0e979b4cf71a6b, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.364545 8448 raft_consensus.cc:2781] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:35.364619 8448 raft_consensus.cc:686] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 1 LEADER]: Becoming Leader. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Running, Role: LEADER
I20260621 05:16:35.364670 8448 consensus_queue.cc:237] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.365222 7932 catalog_manager.cc:5495] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } health_report { overall_health: UNKNOWN } } }
Jun 21 05:16:35 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:35.410533 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.404181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37085 (local address 127.0.0.1:50774)
0621 05:16:35.404350 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.404361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.404423 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.404532 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.404532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.404817 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:35.404858 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.404859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.405405 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.405405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.405900 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.405901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.406192 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.406193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.406228 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.406681 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:35.406684 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:35.408023 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:35.409479 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:35.409486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:35.409489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:35.409662 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:35.409851 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:35.409852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:35.409852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:35.409905 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:35.410144 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:35.410150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:35.410193 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.410276 (+ 83us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.410410 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260621 05:16:35.410544 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.404239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.0.98.62:37085)
0621 05:16:35.404276 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.404279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.404416 (+ 137us) server_negotiation.cc:407] Connection header received
0621 05:16:35.404460 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.404460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.404468 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.404516 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:35.404917 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.405344 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.405967 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.406140 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.406179 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.408118 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:35.408119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:35.408120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:35.408126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:35.409392 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:35.409741 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:35.409742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:35.409742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:35.409789 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:35.409973 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:35.409974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:35.409974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:35.410083 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:35.410153 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.410309 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.410464 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":16}
I20260621 05:16:35.412472 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50774:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:16:35.412688 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:16:35.414009 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 5bda70f91e9c4cfab43b9361ac40efa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b93752704fc404c919887e48cd7f30b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:16:35.414018 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 5bda70f91e9c4cfab43b9361ac40efa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b93752704fc404c919887e48cd7f30b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:16:35.414031 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 5bda70f91e9c4cfab43b9361ac40efa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b93752704fc404c919887e48cd7f30b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:16:35.414155 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bda70f91e9c4cfab43b9361ac40efa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.414155 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bda70f91e9c4cfab43b9361ac40efa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.414177 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bda70f91e9c4cfab43b9361ac40efa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.415264 8443 tablet_bootstrap.cc:492] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: Bootstrap starting.
I20260621 05:16:35.415582 8444 tablet_bootstrap.cc:492] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: Bootstrap starting.
I20260621 05:16:35.415589 8442 tablet_bootstrap.cc:492] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: Bootstrap starting.
I20260621 05:16:35.415695 8443 tablet_bootstrap.cc:654] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.415954 8442 tablet_bootstrap.cc:654] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.415983 8444 tablet_bootstrap.cc:654] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.416441 8442 tablet_bootstrap.cc:492] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: No bootstrap required, opened a new log
I20260621 05:16:35.416479 8442 ts_tablet_manager.cc:1397] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:35.416572 8444 tablet_bootstrap.cc:492] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: No bootstrap required, opened a new log
I20260621 05:16:35.416617 8444 ts_tablet_manager.cc:1397] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:35.416615 8442 raft_consensus.cc:348] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.416711 8442 raft_consensus.cc:374] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.416752 8442 raft_consensus.cc:729] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.416796 8442 consensus_queue.cc:260] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.416791 8444 raft_consensus.cc:348] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.416846 8444 raft_consensus.cc:374] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.416872 8444 raft_consensus.cc:729] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67db0b30d25d480dae0e979b4cf71a6b, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.416879 8442 ts_tablet_manager.cc:1428] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.416893 8448 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.416927 8444 consensus_queue.cc:260] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.416990 8448 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:16:35.417029 8444 ts_tablet_manager.cc:1428] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.417083 8443 tablet_bootstrap.cc:492] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: No bootstrap required, opened a new log
I20260621 05:16:35.417084 8449 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.417129 8443 ts_tablet_manager.cc:1397] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:16:35.417153 8449 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:16:35.417285 8443 raft_consensus.cc:348] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.417364 8443 raft_consensus.cc:374] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.417393 8443 raft_consensus.cc:729] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8e095a154d4405a9468e5513e1140a, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.417443 8443 consensus_queue.cc:260] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.417546 8443 ts_tablet_manager.cc:1428] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.417586 8450 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.417714 8450 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:16:35.473212 8448 raft_consensus.cc:482] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:16:35.473280 8448 raft_consensus.cc:504] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.473489 8448 leader_election.cc:288] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.473676 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bda70f91e9c4cfab43b9361ac40efa9" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8e095a154d4405a9468e5513e1140a" is_pre_election: true
I20260621 05:16:35.473666 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bda70f91e9c4cfab43b9361ac40efa9" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b" is_pre_election: true
I20260621 05:16:35.473774 8231 raft_consensus.cc:2454] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 0.
I20260621 05:16:35.473788 8371 raft_consensus.cc:2454] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 0.
I20260621 05:16:35.473923 8024 leader_election.cc:302] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67db0b30d25d480dae0e979b4cf71a6b, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.474016 8448 raft_consensus.cc:2781] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:16:35.474057 8448 raft_consensus.cc:482] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:16:35.474081 8448 raft_consensus.cc:3037] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.474620 8448 raft_consensus.cc:504] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.474766 8448 leader_election.cc:288] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 election: Requested vote from peers 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.474923 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bda70f91e9c4cfab43b9361ac40efa9" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8e095a154d4405a9468e5513e1140a"
I20260621 05:16:35.474982 8231 raft_consensus.cc:3037] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.475482 8231 raft_consensus.cc:2454] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 1.
I20260621 05:16:35.475593 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bda70f91e9c4cfab43b9361ac40efa9" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b"
I20260621 05:16:35.475651 8024 leader_election.cc:302] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a8e095a154d4405a9468e5513e1140a, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.475697 8371 raft_consensus.cc:3037] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.475808 8448 raft_consensus.cc:2781] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:35.475874 8448 raft_consensus.cc:686] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [term 1 LEADER]: Becoming Leader. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Running, Role: LEADER
I20260621 05:16:35.475919 8448 consensus_queue.cc:237] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.476150 8452 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: TxnStatusTablet state changed. Reason: New leader 6e1223829ee24eaa8930b3a815a3daac. Latest consensus state: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.476246 8452 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:16:35.476420 8481 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:16:35.476483 8371 raft_consensus.cc:2454] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 1.
I20260621 05:16:35.476545 8481 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:16:35.476547 7932 catalog_manager.cc:5495] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } health_report { overall_health: UNKNOWN } } }
I20260621 05:16:35.492841 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.490308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.0.1:48752)
0621 05:16:35.490383 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.490395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.490446 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.490591 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.490592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.490592 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:16:35.490631 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.490632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.491468 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.491468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.492041 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.492043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.492506 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.492507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.492549 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.492561 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.492721 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.492726 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.492759 (+ 33us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.492792 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 05:16:35.492849 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.490364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.0.98.1:43327)
0621 05:16:35.490403 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.490405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.490435 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:16:35.490512 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.490512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.490522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.490587 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:35.490667 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.491403 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.492136 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.492433 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.492503 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.492633 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.492665 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.492716 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.492763 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.492781 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:16:35.493852 8371 raft_consensus.cc:1261] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Refusing update from remote peer 6e1223829ee24eaa8930b3a815a3daac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.493870 8231 raft_consensus.cc:1261] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Refusing update from remote peer 6e1223829ee24eaa8930b3a815a3daac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.494050 8452 consensus_queue.cc:1035] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.494165 8452 consensus_queue.cc:1035] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Connected to new peer: Peer: permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.494562 8450 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: TxnStatusTablet state changed. Reason: New leader 6e1223829ee24eaa8930b3a815a3daac. Latest consensus state: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.494643 8450 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:16:35.494607 8449 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: TxnStatusTablet state changed. Reason: New leader 6e1223829ee24eaa8930b3a815a3daac. Latest consensus state: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.494694 8449 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:16:35.495293 8448 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.495373 8448 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:16:35.495446 8456 mvcc.cc:205] Tried to move back new op lower bound from 7299149805541416960 to 7299149805469605888. Current Snapshot: MvccSnapshot[applied={T|T < 7299149805541416960}]
I20260621 05:16:35.495541 8449 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.495599 8449 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 67db0b30d25d480dae0e979b4cf71a6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:16:35.495302 8455 mvcc.cc:205] Tried to move back new op lower bound from 7299149805541416960 to 7299149805469605888. Current Snapshot: MvccSnapshot[applied={T|T < 7299149805541416960}]
I20260621 05:16:35.496232 8448 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.496307 8448 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 6e1223829ee24eaa8930b3a815a3daac: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:16:35.496781 8457 mvcc.cc:205] Tried to move back new op lower bound from 7299149805541416960 to 7299149805469605888. Current Snapshot: MvccSnapshot[applied={T|T < 7299149805541416960 or (T in {7299149805541416960})}]
I20260621 05:16:35.496915 8450 tablet_replica.cc:439] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } } }
I20260621 05:16:35.496992 8450 tablet_replica.cc:442] T 5bda70f91e9c4cfab43b9361ac40efa9 P 1a8e095a154d4405a9468e5513e1140a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:16:35.501509 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.499141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44199 (local address 127.0.98.1:43327)
0621 05:16:35.499172 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.499174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.499185 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:16:35.499235 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.499235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.499243 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.499300 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:35.499379 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.500101 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.500784 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.501084 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.501153 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.501284 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.501312 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.501363 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.501412 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.501430 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:16:35.501510 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.499024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.98.1:44199)
0621 05:16:35.499147 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.499156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.499196 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.499304 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.499305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.499306 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:16:35.499337 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.499338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.500171 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.500172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.500705 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.500707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.501158 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.501159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.501199 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.501210 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.501374 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.501376 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.501410 (+ 34us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.501440 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":72,"threads_started":1}
I20260621 05:16:35.503829 8371 raft_consensus.cc:1261] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Refusing update from remote peer 6e1223829ee24eaa8930b3a815a3daac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.503866 8231 raft_consensus.cc:1261] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Refusing update from remote peer 6e1223829ee24eaa8930b3a815a3daac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.504012 8489 consensus_queue.cc:1035] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.504132 8452 consensus_queue.cc:1035] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Connected to new peer: Peer: permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.514060 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:35.514151 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:35.514716 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: Sending DeleteTablet for 3 replicas of tablet d9739e97e517478f8dc8a94dcc593941
I20260621 05:16:35.514986 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet d9739e97e517478f8dc8a94dcc593941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46064
I20260621 05:16:35.515004 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet d9739e97e517478f8dc8a94dcc593941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45318
I20260621 05:16:35.515077 8464 tablet_replica.cc:330] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: stopping tablet replica
I20260621 05:16:35.515089 8466 tablet_replica.cc:330] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: stopping tablet replica
I20260621 05:16:35.515117 8464 raft_consensus.cc:2229] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:35.515134 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet d9739e97e517478f8dc8a94dcc593941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48724
I20260621 05:16:35.515151 8466 raft_consensus.cc:2229] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:35.515152 8464 raft_consensus.cc:2258] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.515189 8466 raft_consensus.cc:2258] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.515205 8465 tablet_replica.cc:330] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: stopping tablet replica
I20260621 05:16:35.515259 8465 raft_consensus.cc:2229] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:16:35.515301 8465 raft_consensus.cc:2258] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.515569 8466 ts_tablet_manager.cc:1905] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.515699 8464 ts_tablet_manager.cc:1905] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.517153 8465 ts_tablet_manager.cc:1905] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.517362 8466 ts_tablet_manager.cc:1918] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:16:35.517407 8466 log.cc:1196] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/d9739e97e517478f8dc8a94dcc593941
I20260621 05:16:35.517388 8464 ts_tablet_manager.cc:1918] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:16:35.517444 8464 log.cc:1196] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/d9739e97e517478f8dc8a94dcc593941
I20260621 05:16:35.517608 8466 ts_tablet_manager.cc:1939] T d9739e97e517478f8dc8a94dcc593941 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting consensus metadata
I20260621 05:16:35.517676 8464 ts_tablet_manager.cc:1939] T d9739e97e517478f8dc8a94dcc593941 P 1a8e095a154d4405a9468e5513e1140a: Deleting consensus metadata
I20260621 05:16:35.517875 7920 catalog_manager.cc:4841] TS 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547): tablet d9739e97e517478f8dc8a94dcc593941 (table test-table [id=7a1ae499f10c4ed0973c05648609c403]) successfully deleted
I20260621 05:16:35.517958 7920 catalog_manager.cc:4841] TS 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121): tablet d9739e97e517478f8dc8a94dcc593941 (table test-table [id=7a1ae499f10c4ed0973c05648609c403]) successfully deleted
I20260621 05:16:35.518573 8465 ts_tablet_manager.cc:1918] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:16:35.518610 8465 log.cc:1196] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/d9739e97e517478f8dc8a94dcc593941
I20260621 05:16:35.518744 8465 ts_tablet_manager.cc:1939] T d9739e97e517478f8dc8a94dcc593941 P 6e1223829ee24eaa8930b3a815a3daac: Deleting consensus metadata
I20260621 05:16:35.519130 7919 catalog_manager.cc:4841] TS 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1:43327): tablet d9739e97e517478f8dc8a94dcc593941 (table test-table [id=7a1ae499f10c4ed0973c05648609c403]) successfully deleted
Jun 21 05:16:35 dist-test-slave-czm5 krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:16:35 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:35 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:16:35.540267 7984 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:35.534664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.0.98.62:37085)
0621 05:16:35.534736 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.534738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.534836 (+ 98us) server_negotiation.cc:407] Connection header received
0621 05:16:35.534887 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.534887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.534899 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.534978 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:35.535498 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.536319 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.537042 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.537333 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.537399 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.540185 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50782: BlockingRecv error: recv got EOF from 127.0.0.1:50782 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 21 05:16:35 dist-test-slave-czm5 krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260621 05:16:35.548141 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.541051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.0.98.62:37085)
0621 05:16:35.541085 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.541087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.541189 (+ 102us) server_negotiation.cc:407] Connection header received
0621 05:16:35.541239 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.541240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.541249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.541298 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:35.541699 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.542422 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.543134 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.543381 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.543443 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.545645 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:35.545646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:35.545647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:35.545653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:35.547111 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:35.547419 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:35.547420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:35.547421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:35.547479 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:35.547710 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:35.547711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:35.547711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:35.547828 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:35.547941 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.548002 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.548068 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:16:35.549604 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:16:35.549764 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:16:35.551189 8070 tablet_service.cc:1467] Processing CreateTablet for tablet d5e65af437614233be7c9a2035a9bab2 (DEFAULT_TABLE table=test-table [id=5898dc04f45949759f7a75bbea315332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.551245 8211 tablet_service.cc:1467] Processing CreateTablet for tablet d5e65af437614233be7c9a2035a9bab2 (DEFAULT_TABLE table=test-table [id=5898dc04f45949759f7a75bbea315332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.551324 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e65af437614233be7c9a2035a9bab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.551352 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e65af437614233be7c9a2035a9bab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.552596 8442 tablet_bootstrap.cc:492] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: Bootstrap starting.
I20260621 05:16:35.552649 8351 tablet_service.cc:1467] Processing CreateTablet for tablet d5e65af437614233be7c9a2035a9bab2 (DEFAULT_TABLE table=test-table [id=5898dc04f45949759f7a75bbea315332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.552747 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e65af437614233be7c9a2035a9bab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.552942 8442 tablet_bootstrap.cc:654] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.553184 8443 tablet_bootstrap.cc:492] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: Bootstrap starting.
I20260621 05:16:35.553596 8443 tablet_bootstrap.cc:654] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.553714 8444 tablet_bootstrap.cc:492] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: Bootstrap starting.
I20260621 05:16:35.554062 8442 tablet_bootstrap.cc:492] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: No bootstrap required, opened a new log
I20260621 05:16:35.554112 8442 ts_tablet_manager.cc:1397] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:16:35.554127 8444 tablet_bootstrap.cc:654] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.554212 8442 raft_consensus.cc:348] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.554277 8442 raft_consensus.cc:374] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.554296 8442 raft_consensus.cc:729] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.554332 8442 consensus_queue.cc:260] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.554409 8442 ts_tablet_manager.cc:1428] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.554874 8444 tablet_bootstrap.cc:492] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: No bootstrap required, opened a new log
I20260621 05:16:35.554926 8444 ts_tablet_manager.cc:1397] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:35.555068 8443 tablet_bootstrap.cc:492] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: No bootstrap required, opened a new log
I20260621 05:16:35.555058 8444 raft_consensus.cc:348] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.555101 8444 raft_consensus.cc:374] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.555114 8444 raft_consensus.cc:729] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67db0b30d25d480dae0e979b4cf71a6b, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.555111 8443 ts_tablet_manager.cc:1397] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:16:35.555171 8444 consensus_queue.cc:260] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.555256 8444 ts_tablet_manager.cc:1428] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.555250 8443 raft_consensus.cc:348] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.555307 8443 raft_consensus.cc:374] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.555326 8443 raft_consensus.cc:729] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8e095a154d4405a9468e5513e1140a, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.555363 8443 consensus_queue.cc:260] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.555438 8443 ts_tablet_manager.cc:1428] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.817288 8489 raft_consensus.cc:482] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:16:35.817381 8489 raft_consensus.cc:504] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.817569 8489 leader_election.cc:288] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.817814 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e65af437614233be7c9a2035a9bab2" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b" is_pre_election: true
I20260621 05:16:35.817921 8371 raft_consensus.cc:2454] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 0.
I20260621 05:16:35.818045 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e65af437614233be7c9a2035a9bab2" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8e095a154d4405a9468e5513e1140a" is_pre_election: true
I20260621 05:16:35.818078 8024 leader_election.cc:302] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67db0b30d25d480dae0e979b4cf71a6b, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.818128 8231 raft_consensus.cc:2454] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 0.
I20260621 05:16:35.818168 8489 raft_consensus.cc:2781] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:16:35.818202 8489 raft_consensus.cc:482] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:16:35.818233 8489 raft_consensus.cc:3037] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.818969 8489 raft_consensus.cc:504] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.819095 8489 leader_election.cc:288] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 election: Requested vote from peers 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.819234 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e65af437614233be7c9a2035a9bab2" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8e095a154d4405a9468e5513e1140a"
I20260621 05:16:35.819283 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e65af437614233be7c9a2035a9bab2" candidate_uuid: "6e1223829ee24eaa8930b3a815a3daac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b"
I20260621 05:16:35.819319 8231 raft_consensus.cc:3037] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.819344 8371 raft_consensus.cc:3037] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.820055 8231 raft_consensus.cc:2454] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 1.
I20260621 05:16:35.820055 8371 raft_consensus.cc:2454] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1223829ee24eaa8930b3a815a3daac in term 1.
I20260621 05:16:35.820256 8024 leader_election.cc:302] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a8e095a154d4405a9468e5513e1140a, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.820369 8489 raft_consensus.cc:2781] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:35.820426 8489 raft_consensus.cc:686] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 1 LEADER]: Becoming Leader. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Running, Role: LEADER
I20260621 05:16:35.820499 8489 consensus_queue.cc:237] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.821027 7932 catalog_manager.cc:5495] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6e1223829ee24eaa8930b3a815a3daac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } health_report { overall_health: UNKNOWN } } }
I20260621 05:16:35.837563 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.835538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.98.1:43327)
0621 05:16:35.835594 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.835595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.835603 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:35.835700 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.835700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.835709 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.835757 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:35.835862 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.836340 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.837039 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.837204 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.837251 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.837369 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.837392 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.837429 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.837460 (+ 31us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.837473 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:16:35.838339 8371 raft_consensus.cc:1261] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Refusing update from remote peer 6e1223829ee24eaa8930b3a815a3daac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.838341 8231 raft_consensus.cc:1261] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Refusing update from remote peer 6e1223829ee24eaa8930b3a815a3daac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.838567 8452 consensus_queue.cc:1035] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Connected to new peer: Peer: permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.838683 8489 consensus_queue.cc:1035] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.841046 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:35.841131 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:35.841719 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: Sending DeleteTablet for 3 replicas of tablet d5e65af437614233be7c9a2035a9bab2
I20260621 05:16:35.841987 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e65af437614233be7c9a2035a9bab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48724
I20260621 05:16:35.842036 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e65af437614233be7c9a2035a9bab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46064
I20260621 05:16:35.842079 8465 tablet_replica.cc:330] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: stopping tablet replica
I20260621 05:16:35.842082 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e65af437614233be7c9a2035a9bab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45318
I20260621 05:16:35.842128 8465 raft_consensus.cc:2229] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:16:35.842165 8466 tablet_replica.cc:330] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: stopping tablet replica
I20260621 05:16:35.842172 8464 tablet_replica.cc:330] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: stopping tablet replica
I20260621 05:16:35.842182 8465 raft_consensus.cc:2258] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.842216 8466 raft_consensus.cc:2229] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:35.842223 8464 raft_consensus.cc:2229] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:35.842170 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 05:16:35.842252 8466 raft_consensus.cc:2258] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.842262 8464 raft_consensus.cc:2258] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 05:16:35.842301 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:16:35.842644 8464 ts_tablet_manager.cc:1905] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.842667 8465 ts_tablet_manager.cc:1905] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.844240 8464 ts_tablet_manager.cc:1918] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:35.844264 8465 ts_tablet_manager.cc:1918] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:35.844300 8464 log.cc:1196] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/d5e65af437614233be7c9a2035a9bab2
I20260621 05:16:35.844316 8465 log.cc:1196] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/d5e65af437614233be7c9a2035a9bab2
I20260621 05:16:35.844496 8465 ts_tablet_manager.cc:1939] T d5e65af437614233be7c9a2035a9bab2 P 6e1223829ee24eaa8930b3a815a3daac: Deleting consensus metadata
I20260621 05:16:35.844540 8464 ts_tablet_manager.cc:1939] T d5e65af437614233be7c9a2035a9bab2 P 1a8e095a154d4405a9468e5513e1140a: Deleting consensus metadata
I20260621 05:16:35.844739 7919 catalog_manager.cc:4841] TS 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1:43327): tablet d5e65af437614233be7c9a2035a9bab2 (table test-table [id=5898dc04f45949759f7a75bbea315332]) successfully deleted
I20260621 05:16:35.844924 8466 ts_tablet_manager.cc:1905] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.844988 7920 catalog_manager.cc:4841] TS 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121): tablet d5e65af437614233be7c9a2035a9bab2 (table test-table [id=5898dc04f45949759f7a75bbea315332]) successfully deleted
I20260621 05:16:35.845371 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2682b05f304309bce41757ffac02d1 (DEFAULT_TABLE table=test-table [id=c92b88ae809040d28b6691f0bf5394f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.845386 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2682b05f304309bce41757ffac02d1 (DEFAULT_TABLE table=test-table [id=c92b88ae809040d28b6691f0bf5394f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.845470 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2682b05f304309bce41757ffac02d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.845470 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2682b05f304309bce41757ffac02d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.846148 8466 ts_tablet_manager.cc:1918] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:35.846185 8466 log.cc:1196] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/d5e65af437614233be7c9a2035a9bab2
I20260621 05:16:35.846329 8466 ts_tablet_manager.cc:1939] T d5e65af437614233be7c9a2035a9bab2 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting consensus metadata
I20260621 05:16:35.846561 7920 catalog_manager.cc:4841] TS 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547): tablet d5e65af437614233be7c9a2035a9bab2 (table test-table [id=5898dc04f45949759f7a75bbea315332]) successfully deleted
I20260621 05:16:35.846608 8443 tablet_bootstrap.cc:492] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: Bootstrap starting.
I20260621 05:16:35.846664 8442 tablet_bootstrap.cc:492] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac: Bootstrap starting.
I20260621 05:16:35.846649 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2682b05f304309bce41757ffac02d1 (DEFAULT_TABLE table=test-table [id=c92b88ae809040d28b6691f0bf5394f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:35.846745 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2682b05f304309bce41757ffac02d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:35.846930 8442 tablet_bootstrap.cc:654] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.846942 8443 tablet_bootstrap.cc:654] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.847747 8442 tablet_bootstrap.cc:492] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac: No bootstrap required, opened a new log
I20260621 05:16:35.847802 8442 ts_tablet_manager.cc:1397] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:35.847867 8443 tablet_bootstrap.cc:492] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: No bootstrap required, opened a new log
I20260621 05:16:35.847898 8443 ts_tablet_manager.cc:1397] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:35.847906 8442 raft_consensus.cc:348] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.848002 8442 raft_consensus.cc:374] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.848026 8442 raft_consensus.cc:729] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1223829ee24eaa8930b3a815a3daac, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.848026 8443 raft_consensus.cc:348] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.848062 8444 tablet_bootstrap.cc:492] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: Bootstrap starting.
I20260621 05:16:35.848083 8443 raft_consensus.cc:374] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.848064 8442 consensus_queue.cc:260] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.848105 8443 raft_consensus.cc:729] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8e095a154d4405a9468e5513e1140a, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.848155 8442 ts_tablet_manager.cc:1428] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.848147 8443 consensus_queue.cc:260] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.848232 8443 ts_tablet_manager.cc:1428] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.848457 8444 tablet_bootstrap.cc:654] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:35.849291 8444 tablet_bootstrap.cc:492] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: No bootstrap required, opened a new log
I20260621 05:16:35.849351 8444 ts_tablet_manager.cc:1397] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:35.849493 8444 raft_consensus.cc:348] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.849541 8444 raft_consensus.cc:374] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:35.849583 8444 raft_consensus.cc:729] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67db0b30d25d480dae0e979b4cf71a6b, State: Initialized, Role: FOLLOWER
I20260621 05:16:35.849633 8444 consensus_queue.cc:260] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.849716 8444 ts_tablet_manager.cc:1428] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:35.871814 8450 raft_consensus.cc:482] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:16:35.871878 8450 raft_consensus.cc:504] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.872210 8450 leader_election.cc:288] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1:43327), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.875216 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.872390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59379 (local address 127.0.98.1:43327)
0621 05:16:35.872434 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.872436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.872517 (+ 81us) server_negotiation.cc:407] Connection header received
0621 05:16:35.872579 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.872580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.872588 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.872666 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:35.872827 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.873357 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.874467 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.874860 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.874910 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.874919 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.875119 (+ 200us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.875152 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:16:35.875236 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.872341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36547 (local address 127.0.98.2:36125)
0621 05:16:35.872464 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.872474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.872524 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.872670 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.872671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.872673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:35.872775 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.872776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.873583 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.873584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.874388 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.874391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.874979 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.874979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.875045 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.875048 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.875113 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.875152 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":35,"thread_start_us":45,"threads_started":1}
I20260621 05:16:35.875240 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.872404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36125 (local address 127.0.98.3:36547)
0621 05:16:35.872503 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.872505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.872514 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:35.872580 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.872580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.872591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.872658 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:35.872882 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.873487 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.874535 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.874876 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.874971 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.874994 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.875118 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.875153 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":55,"threads_started":1}
I20260621 05:16:35.875253 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.872340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.98.2:59379)
0621 05:16:35.872452 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.872462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.872524 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.872671 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.872672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.872674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:35.872758 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.872759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.873461 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.873462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.874367 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.874369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.874957 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.874958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.875036 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.875039 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.875112 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.875159 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":5,"thread_start_us":52,"threads_started":1}
I20260621 05:16:35.875588 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2682b05f304309bce41757ffac02d1" candidate_uuid: "1a8e095a154d4405a9468e5513e1140a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1223829ee24eaa8930b3a815a3daac" is_pre_election: true
I20260621 05:16:35.875725 8091 raft_consensus.cc:2454] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a8e095a154d4405a9468e5513e1140a in term 0.
I20260621 05:16:35.875835 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2682b05f304309bce41757ffac02d1" candidate_uuid: "1a8e095a154d4405a9468e5513e1140a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b" is_pre_election: true
I20260621 05:16:35.875918 8371 raft_consensus.cc:2454] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a8e095a154d4405a9468e5513e1140a in term 0.
I20260621 05:16:35.875902 8163 leader_election.cc:302] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a8e095a154d4405a9468e5513e1140a, 6e1223829ee24eaa8930b3a815a3daac; no voters:
I20260621 05:16:35.876042 8450 raft_consensus.cc:2781] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:16:35.876099 8450 raft_consensus.cc:482] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:16:35.876123 8450 raft_consensus.cc:3037] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.876689 8450 raft_consensus.cc:504] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.876816 8450 leader_election.cc:288] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [CANDIDATE]: Term 1 election: Requested vote from peers 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1:43327), 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547)
I20260621 05:16:35.876948 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2682b05f304309bce41757ffac02d1" candidate_uuid: "1a8e095a154d4405a9468e5513e1140a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67db0b30d25d480dae0e979b4cf71a6b"
I20260621 05:16:35.877009 8371 raft_consensus.cc:3037] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.877552 8371 raft_consensus.cc:2454] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a8e095a154d4405a9468e5513e1140a in term 1.
I20260621 05:16:35.877656 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2682b05f304309bce41757ffac02d1" candidate_uuid: "1a8e095a154d4405a9468e5513e1140a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1223829ee24eaa8930b3a815a3daac"
I20260621 05:16:35.877722 8091 raft_consensus.cc:3037] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:35.877700 8164 leader_election.cc:302] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a8e095a154d4405a9468e5513e1140a, 67db0b30d25d480dae0e979b4cf71a6b; no voters:
I20260621 05:16:35.877794 8450 raft_consensus.cc:2781] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:35.877904 8450 raft_consensus.cc:686] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 1 LEADER]: Becoming Leader. State: Replica: 1a8e095a154d4405a9468e5513e1140a, State: Running, Role: LEADER
I20260621 05:16:35.877976 8450 consensus_queue.cc:237] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } }
I20260621 05:16:35.878542 8091 raft_consensus.cc:2454] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a8e095a154d4405a9468e5513e1140a in term 1.
I20260621 05:16:35.878587 7934 catalog_manager.cc:5495] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a8e095a154d4405a9468e5513e1140a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1a8e095a154d4405a9468e5513e1140a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a8e095a154d4405a9468e5513e1140a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 } health_report { overall_health: UNKNOWN } } }
I20260621 05:16:35.886863 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.884372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.0.98.2:33121)
0621 05:16:35.884588 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.884589 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.884594 (+ 5us) server_negotiation.cc:407] Connection header received
0621 05:16:35.884617 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.884618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.884630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.884705 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:35.884802 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.885365 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.886080 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.886310 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.886394 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.886556 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.886655 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.886730 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.886765 (+ 35us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.886781 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260621 05:16:35.891134 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.888915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50307 (local address 127.0.98.1:43327)
0621 05:16:35.888969 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.888970 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.889083 (+ 113us) server_negotiation.cc:407] Connection header received
0621 05:16:35.889142 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.889143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.889153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.889204 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:35.889315 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.889714 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.890468 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.890647 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.890726 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.890898 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.890933 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.890992 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.891052 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.891068 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:16:35.891178 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.888857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.98.2:50307)
0621 05:16:35.889013 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.889030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.889096 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.889207 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.889208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.889209 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:16:35.889256 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.889257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.889788 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.889789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.890372 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.890374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.890747 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.890748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.890803 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.890818 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.890994 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.890996 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.891047 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.891080 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260621 05:16:35.892848 8371 raft_consensus.cc:1261] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Refusing update from remote peer 1a8e095a154d4405a9468e5513e1140a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.892940 8091 raft_consensus.cc:1261] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Refusing update from remote peer 1a8e095a154d4405a9468e5513e1140a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:35.893050 8450 consensus_queue.cc:1035] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67db0b30d25d480dae0e979b4cf71a6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.893169 8518 consensus_queue.cc:1035] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1223829ee24eaa8930b3a815a3daac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:35.899176 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.897163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39157 (local address 127.0.98.2:33121)
0621 05:16:35.897198 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:35.897200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:35.897206 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:16:35.897272 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:35.897273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:35.897280 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:35.897330 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:35.897423 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.897839 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.898501 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.898693 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.898763 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.898916 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.898945 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.898988 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:35.899091 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 05:16:35.899109 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:16:35.899176 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.897131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33121 (local address 127.0.98.1:39157)
0621 05:16:35.897173 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.897184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:35.897217 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:35.897334 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:35.897334 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:35.897335 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:16:35.897372 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.897373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.897934 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.897935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.898406 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:35.898408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:35.898791 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:35.898792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:35.898834 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:35.898838 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:16:35.899037 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:16:35.899039 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:16:35.899077 (+ 38us) client_negotiation.cc:240] Negotiation successful
0621 05:16:35.899117 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:16:35.905401 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:35.905493 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:35.905979 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23e3437c9b31449fae200708d2356446: Sending DeleteTablet for 3 replicas of tablet 4c2682b05f304309bce41757ffac02d1
I20260621 05:16:35.906167 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2682b05f304309bce41757ffac02d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46064
I20260621 05:16:35.906188 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2682b05f304309bce41757ffac02d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48724
I20260621 05:16:35.906219 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2682b05f304309bce41757ffac02d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45318
I20260621 05:16:35.906243 8464 tablet_replica.cc:330] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: stopping tablet replica
I20260621 05:16:35.906260 8465 tablet_replica.cc:330] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac: stopping tablet replica
I20260621 05:16:35.906287 8464 raft_consensus.cc:2229] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:16:35.906297 8465 raft_consensus.cc:2229] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:35.906324 8465 raft_consensus.cc:2258] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.906363 8464 raft_consensus.cc:2258] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.906389 8466 tablet_replica.cc:330] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: stopping tablet replica
I20260621 05:16:35.906436 8466 raft_consensus.cc:2229] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:35.906474 8466 raft_consensus.cc:2258] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:35.906854 8465 ts_tablet_manager.cc:1905] T 4c2682b05f304309bce41757ffac02d1 P 6e1223829ee24eaa8930b3a815a3daac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.907904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 8006
I20260621 05:16:35.908047 8466 ts_tablet_manager.cc:1905] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.908985 8464 ts_tablet_manager.cc:1905] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:35.909607 8466 ts_tablet_manager.cc:1918] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:16:35.909655 8466 log.cc:1196] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/4c2682b05f304309bce41757ffac02d1
I20260621 05:16:35.909861 8466 ts_tablet_manager.cc:1939] T 4c2682b05f304309bce41757ffac02d1 P 67db0b30d25d480dae0e979b4cf71a6b: Deleting consensus metadata
I20260621 05:16:35.910293 8464 ts_tablet_manager.cc:1918] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:16:35.910349 8464 log.cc:1196] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/4c2682b05f304309bce41757ffac02d1
I20260621 05:16:35.910507 7920 catalog_manager.cc:4841] TS 67db0b30d25d480dae0e979b4cf71a6b (127.0.98.3:36547): tablet 4c2682b05f304309bce41757ffac02d1 (table test-table [id=c92b88ae809040d28b6691f0bf5394f6]) successfully deleted
I20260621 05:16:35.910549 8464 ts_tablet_manager.cc:1939] T 4c2682b05f304309bce41757ffac02d1 P 1a8e095a154d4405a9468e5513e1140a: Deleting consensus metadata
I20260621 05:16:35.910923 7920 catalog_manager.cc:4841] TS 1a8e095a154d4405a9468e5513e1140a (127.0.98.2:33121): tablet 4c2682b05f304309bce41757ffac02d1 (table test-table [id=c92b88ae809040d28b6691f0bf5394f6]) successfully deleted
W20260621 05:16:35.917537 7919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43327 (error 108)
I20260621 05:16:35.917834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 8146
I20260621 05:16:35.917996 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:35.917831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.0.1:48770)
0621 05:16:35.917920 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:35.917958 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":56,"threads_started":1}
W20260621 05:16:35.919365 7919 catalog_manager.cc:4584] TS 6e1223829ee24eaa8930b3a815a3daac (127.0.98.1:43327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c2682b05f304309bce41757ffac02d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43327: connect: Connection refused (error 111)
I20260621 05:16:35.922116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 8286
I20260621 05:16:35.926293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 7902
2026-06-21T05:16:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2662 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:16:35 dist-test-slave-czm5 krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:16:35 dist-test-slave-czm5 krb5kdc[8532](info): set up 2 sockets
Jun 21 05:16:35 dist-test-slave-czm5 krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260621 05:16:36.932096 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:16:37 dist-test-slave-czm5 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:16:37.037659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:37.110977 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:37.111155 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:37.111176 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:37.112357 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:37.112396 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:37.112409 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:37.112421 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:37.112473 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:37.113917 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:37.114148 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:37.114384 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:37.118829 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:37.118899 8545 server_base.cc:1034] running on GCE node
W20260621 05:16:37.118827 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:37.118844 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:37.119216 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:37.119486 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:37.120627 8545 hybrid_clock.cc:648] HybridClock initialized: now 1782018997120593 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T05:16:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1198 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:16:37 dist-test-slave-czm5 krb5kdc[8558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:16:37 dist-test-slave-czm5 krb5kdc[8558](info): set up 2 sockets
Jun 21 05:16:37 dist-test-slave-czm5 krb5kdc[8558](info): commencing operation
krb5kdc: starting...
W20260621 05:16:38.124320 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:16:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:16:38.234097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:38.309247 8571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:38.309474 8571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:38.309509 8571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:38.310601 8571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:38.310668 8571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:38.310694 8571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:38.310715 8571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:38.310743 8571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:38.312278 8571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:38.312536 8571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:38.312736 8571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:38.317269 8576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:38.317262 8579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:38.317262 8577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:38.317497 8571 server_base.cc:1034] running on GCE node
I20260621 05:16:38.317759 8571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:38.318033 8571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:38.319165 8571 hybrid_clock.cc:648] HybridClock initialized: now 1782018998319153 us; error 27 us; skew 500 ppm
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:38.324633 8571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:16:38.325076 8571 webserver.cc:460] Webserver started at http://127.0.98.62:43119/ using document root <none> and password file <none>
I20260621 05:16:38.325284 8571 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:38.325348 8571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:38.325495 8571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:38.326515 8571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "7eaf21eb6156485f89b752897c5b8a90"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "cf4e6a95f05fe4fc587f6d84c4fc7f19"
server_key_iv: "f6157f5296f7cc5a276570ba6bb43554"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.326853 8571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "7eaf21eb6156485f89b752897c5b8a90"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "cf4e6a95f05fe4fc587f6d84c4fc7f19"
server_key_iv: "f6157f5296f7cc5a276570ba6bb43554"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.328222 8571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:38.328907 8586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.329064 8571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:38.329159 8571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "7eaf21eb6156485f89b752897c5b8a90"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "cf4e6a95f05fe4fc587f6d84c4fc7f19"
server_key_iv: "f6157f5296f7cc5a276570ba6bb43554"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.329232 8571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:38.350502 8571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:38.352892 8571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:38.353072 8571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:38.357008 8571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43773
I20260621 05:16:38.357043 8638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43773 every 8 connection(s)
I20260621 05:16:38.357513 8571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:16:38.358155 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:38.358464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 8571
I20260621 05:16:38.358542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:16:38.358722 392 external_mini_cluster.cc:1428] Setting key e56440bfda75ced6725547aeeed65533
I20260621 05:16:38.361589 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Bootstrap starting.
I20260621 05:16:38.362263 8639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:38.362563 8639 log.cc:824] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:38.363233 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: No bootstrap required, opened a new log
I20260621 05:16:38.364998 8639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260621 05:16:38.365147 8639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:38.365200 8639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eaf21eb6156485f89b752897c5b8a90, State: Initialized, Role: FOLLOWER
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:38.365358 8639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260621 05:16:38.365447 8639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:16:38.365486 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:16:38.365532 8639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:38.366091 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260621 05:16:38.366195 8639 leader_election.cc:302] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eaf21eb6156485f89b752897c5b8a90; no voters:
I20260621 05:16:38.366343 8639 leader_election.cc:288] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:16:38.366405 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:38.366631 8639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:38.366640 8644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 1 LEADER]: Becoming Leader. State: Replica: 7eaf21eb6156485f89b752897c5b8a90, State: Running, Role: LEADER
I20260621 05:16:38.366812 8644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260621 05:16:38.367233 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eaf21eb6156485f89b752897c5b8a90. Latest consensus state: current_term: 1 leader_uuid: "7eaf21eb6156485f89b752897c5b8a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } } }
I20260621 05:16:38.367233 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eaf21eb6156485f89b752897c5b8a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } } }
I20260621 05:16:38.367414 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:38.367403 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:38.367976 8653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:16:38.368515 8653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:16:38.369761 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:38.359246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.0.98.62:43773)
0621 05:16:38.359446 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:38.359455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:38.359489 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:16:38.359975 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:38.359984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:38.360045 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:38.360220 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:38.360817 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.362167 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.363040 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.363470 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.363582 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:38.365893 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:38.365897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:38.365897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:38.365908 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:38.368081 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:38.368544 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:38.368547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:38.368547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:38.368648 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:38.368869 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:38.368870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:38.368870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:38.369166 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:38.369321 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:38.369432 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 05:16:38.369563 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260621 05:16:38.370132 8653 catalog_manager.cc:1347] Generated new cluster ID: 9d245cf9d4a4409e9da9972d351be45e
I20260621 05:16:38.370170 8653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:16:38.374095 8653 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:16:38.374763 8653 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:16:38.379578 8653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Generated new TSK 0
I20260621 05:16:38.379761 8653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:16:38.410120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43773
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:38.493718 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:38.493983 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:38.494009 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:38.495183 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:38.495223 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:38.495249 8665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:16:38.496735 8665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:38.496968 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:38.497200 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:16:38.502074 8665 server_base.cc:1034] running on GCE node
W20260621 05:16:38.502053 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:38.502070 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:38.502050 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:38.502441 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:38.502682 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:38.503832 8665 hybrid_clock.cc:648] HybridClock initialized: now 1782018998503814 us; error 37 us; skew 500 ppm
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:38.506328 8665 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:16:38.506747 8665 webserver.cc:460] Webserver started at http://127.0.98.1:34137/ using document root <none> and password file <none>
I20260621 05:16:38.506915 8665 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:38.506953 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:38.507041 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:38.508049 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "2e04701f4100474fbd0c7e656aaf126f"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "af7e0d1b2747544acdcf0ed9e5d398c8"
server_key_iv: "0986738a3594ee3bd72e6b05a62efdba"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.508339 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2e04701f4100474fbd0c7e656aaf126f"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "af7e0d1b2747544acdcf0ed9e5d398c8"
server_key_iv: "0986738a3594ee3bd72e6b05a62efdba"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.509543 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:16:38.510249 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.510419 8665 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:38.510533 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "2e04701f4100474fbd0c7e656aaf126f"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "af7e0d1b2747544acdcf0ed9e5d398c8"
server_key_iv: "0986738a3594ee3bd72e6b05a62efdba"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.510605 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:38.522900 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:38.525069 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:38.525224 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:38.525434 8665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:38.525729 8665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:38.525782 8665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.525820 8665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:38.525847 8665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.531354 8665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35643
I20260621 05:16:38.531401 8793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35643 every 8 connection(s)
I20260621 05:16:38.531769 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:16:38.534451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 8665
I20260621 05:16:38.534565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:16:38.534760 392 external_mini_cluster.cc:1428] Setting key 855427310d6d7e60e7e524f3cff9b2e2
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:38.541131 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:38.532384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60599 (local address 127.0.98.62:43773)
0621 05:16:38.532442 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:38.532444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:38.532644 (+ 200us) server_negotiation.cc:407] Connection header received
0621 05:16:38.533553 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:38.533554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:38.533566 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:38.533632 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:38.534350 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.535012 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.536059 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.536281 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.536354 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:38.538367 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:38.538369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:38.538369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:38.538376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:38.539764 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:38.540160 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:38.540161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:38.540162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:38.540217 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:38.540500 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:38.540501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:38.540501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:38.540617 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:38.540696 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:38.540952 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 05:16:38.541045 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:16:38.541298 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:38.532389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:60599)
0621 05:16:38.532539 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:38.532569 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:38.533432 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:38.533744 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:38.533747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:38.534016 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:38.534258 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:38.534263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.535102 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.535103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:38.535975 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:38.535977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.536371 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.536372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:38.536493 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:38.536999 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:38.537006 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:38.538276 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:38.539852 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:38.539858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:38.539860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:38.540074 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:38.540298 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:38.540299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:38.540299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:38.540417 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:38.540685 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:38.540691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:38.540823 (+ 132us) client_negotiation.cc:769] Sending connection context
0621 05:16:38.540930 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:16:38.541074 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260621 05:16:38.541777 8794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43773
I20260621 05:16:38.541864 8794 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:38.542073 8794 heartbeater.cc:511] Master 127.0.98.62:43773 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260621 05:16:38.542579 8603 ts_manager.cc:194] Registered new tserver with Master: 2e04701f4100474fbd0c7e656aaf126f (127.0.98.1:35643)
I20260621 05:16:38.543143 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:16:38.557904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43773
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:38.635604 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:38.635802 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:38.635824 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:38.636893 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:38.636926 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:38.636951 8799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:38.638350 8799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:38.638513 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:38.638675 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:38.643441 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:38.643448 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:38.643448 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:38.643697 8799 server_base.cc:1034] running on GCE node
I20260621 05:16:38.643908 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:38.644137 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:38.645283 8799 hybrid_clock.cc:648] HybridClock initialized: now 1782018998645271 us; error 25 us; skew 500 ppm
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:38.647574 8799 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:16:38.648029 8799 webserver.cc:460] Webserver started at http://127.0.98.2:33755/ using document root <none> and password file <none>
I20260621 05:16:38.648252 8799 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:38.648296 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:38.648438 8799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:38.649412 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "bb0357d93a62424bb7c393467a78a974"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "8c8df475a5747ab6b719cc2186b03aac"
server_key_iv: "4a574acb76e7385cef5790076f370ea5"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.649748 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bb0357d93a62424bb7c393467a78a974"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "8c8df475a5747ab6b719cc2186b03aac"
server_key_iv: "4a574acb76e7385cef5790076f370ea5"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.650986 8799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:16:38.651677 8814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.651829 8799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:38.651891 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "bb0357d93a62424bb7c393467a78a974"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "8c8df475a5747ab6b719cc2186b03aac"
server_key_iv: "4a574acb76e7385cef5790076f370ea5"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.652004 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:38.675189 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:38.676877 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:38.677026 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:38.677225 8799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:38.677516 8799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:38.677567 8799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.677598 8799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:38.677623 8799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.683056 8799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34591
I20260621 05:16:38.683089 8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34591 every 8 connection(s)
I20260621 05:16:38.683385 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:38.691908 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:38.684079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43929 (local address 127.0.98.62:43773)
0621 05:16:38.684166 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:38.684168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:38.684316 (+ 148us) server_negotiation.cc:407] Connection header received
0621 05:16:38.685178 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:38.685180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:38.685191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:38.685237 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:38.685799 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.686246 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.687194 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.687398 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.687469 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:38.689431 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:38.689432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:38.689432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:38.689439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:38.690565 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:38.690957 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:38.690958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:38.690958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:38.691015 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:38.691257 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:38.691257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:38.691258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:38.691382 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:38.691457 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:38.691747 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 05:16:38.691821 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:16:38.692081 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:38.684043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:43929)
0621 05:16:38.684201 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:38.684238 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:38.685067 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:38.685321 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:38.685325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:38.685555 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:38.685711 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:38.685715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.686339 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.686340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:38.687096 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:38.687098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.687474 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.687475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:38.687590 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:38.688073 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:38.688081 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:38.689328 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:38.690647 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:38.690653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:38.690655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:38.690860 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:38.691079 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:38.691081 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:38.691081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:38.691180 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:38.691448 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:38.691454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:38.691594 (+ 140us) client_negotiation.cc:769] Sending connection context
0621 05:16:38.691727 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 05:16:38.691867 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260621 05:16:38.692144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 8799
I20260621 05:16:38.692250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:16:38.692416 392 external_mini_cluster.cc:1428] Setting key a6a7de5f8f5e509c9d33e60bac9a1086
I20260621 05:16:38.692484 8928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43773
I20260621 05:16:38.692567 8928 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:38.692739 8928 heartbeater.cc:511] Master 127.0.98.62:43773 requested a full tablet report, sending...
I20260621 05:16:38.693087 8603 ts_manager.cc:194] Registered new tserver with Master: bb0357d93a62424bb7c393467a78a974 (127.0.98.2:34591)
I20260621 05:16:38.693425 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43929
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:16:38.714666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43773
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:38.791108 8933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:38.791322 8933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:38.791359 8933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:38.792520 8933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:38.792582 8933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:38.792619 8933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:38.794030 8933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:38.794296 8933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:38.794516 8933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:38.799160 8938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:38.799181 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:38.799167 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:38.799415 8933 server_base.cc:1034] running on GCE node
I20260621 05:16:38.799608 8933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:38.799916 8933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:38.801061 8933 hybrid_clock.cc:648] HybridClock initialized: now 1782018998801045 us; error 33 us; skew 500 ppm
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:38.803297 8933 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:16:38.803761 8933 webserver.cc:460] Webserver started at http://127.0.98.3:44811/ using document root <none> and password file <none>
I20260621 05:16:38.803985 8933 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:38.804028 8933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:38.804159 8933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:38.805127 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa805e8a26bb4549b5ca8501a815615e"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "99f772ce3de38db4049a2b75bf827266"
server_key_iv: "234d8f13b4515dd0c544f1d6af411060"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.805507 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa805e8a26bb4549b5ca8501a815615e"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "99f772ce3de38db4049a2b75bf827266"
server_key_iv: "234d8f13b4515dd0c544f1d6af411060"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.806777 8933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:38.807413 8948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.807595 8933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:38.807713 8933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "fa805e8a26bb4549b5ca8501a815615e"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "99f772ce3de38db4049a2b75bf827266"
server_key_iv: "234d8f13b4515dd0c544f1d6af411060"
server_key_version: "encryptionkey@0"
I20260621 05:16:38.807791 8933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:38.816908 8933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:38.818423 8933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:38.818567 8933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:38.818765 8933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:38.819036 8933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:38.819085 8933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.819114 8933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:38.819139 8933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:38.824822 8933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34123
I20260621 05:16:38.824860 9061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34123 every 8 connection(s)
I20260621 05:16:38.825168 8933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:16:38.828758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 8933
I20260621 05:16:38.828884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:16:38.829067 392 external_mini_cluster.cc:1428] Setting key b3dd58e417c9a79e2eb0015f95a8584c
Jun 21 05:16:38 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:38.833879 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:38.825857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57463 (local address 127.0.98.62:43773)
0621 05:16:38.825921 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:38.825923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:38.826089 (+ 166us) server_negotiation.cc:407] Connection header received
0621 05:16:38.826939 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:38.826940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:38.826949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:38.827002 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:38.827556 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.828091 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.829136 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.829387 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.829460 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:38.831537 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:38.831538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:38.831538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:38.831544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:38.832834 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:38.833175 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:38.833175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:38.833176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:38.833222 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:38.833416 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:38.833417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:38.833417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:38.833508 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:38.833555 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:38.833760 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 05:16:38.833813 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:16:38.834056 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:38.825829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:57463)
0621 05:16:38.825967 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:38.826003 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:38.826855 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:38.827053 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:38.827056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:38.827277 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:38.827454 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:38.827458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.828164 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.828165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:38.829029 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:38.829031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:38.829478 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:38.829479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:38.829623 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:38.830149 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:38.830163 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:38.831447 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:38.832910 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:38.832915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:38.832917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:38.833116 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:38.833279 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:38.833280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:38.833280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:38.833367 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:38.833559 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:38.833563 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:38.833660 (+ 97us) client_negotiation.cc:769] Sending connection context
0621 05:16:38.833736 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:16:38.833841 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260621 05:16:38.834453 9062 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43773
I20260621 05:16:38.834576 9062 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:38.834757 9062 heartbeater.cc:511] Master 127.0.98.62:43773 requested a full tablet report, sending...
I20260621 05:16:38.835135 8603 ts_manager.cc:194] Registered new tserver with Master: fa805e8a26bb4549b5ca8501a815615e (127.0.98.3:34123)
I20260621 05:16:38.835512 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57463
I20260621 05:16:38.840762 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260621 05:16:38.901845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 8665
I20260621 05:16:38.906085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 8799
I20260621 05:16:38.910244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 8933
I20260621 05:16:38.914251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 8571
I20260621 05:16:38.919206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43773
--webserver_interface=127.0.98.62
--webserver_port=43119
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:38.996913 9071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:38.997088 9071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:38.997108 9071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:38.998102 9071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:38.998133 9071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:38.998147 9071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:38.998159 9071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:38.998183 9071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:38.999541 9071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:38.999784 9071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:39.000005 9071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:39.004521 9076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:39.004546 9079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:39.004557 9077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:39.004791 9071 server_base.cc:1034] running on GCE node
I20260621 05:16:39.005044 9071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:39.005362 9071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:39.006531 9071 hybrid_clock.cc:648] HybridClock initialized: now 1782018999006525 us; error 34 us; skew 500 ppm
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:39.008826 9071 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:16:39.009215 9071 webserver.cc:460] Webserver started at http://127.0.98.62:43119/ using document root <none> and password file <none>
I20260621 05:16:39.009436 9071 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:39.009514 9071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:39.010740 9071 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:39.011334 9086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.011495 9071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:39.011608 9071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "7eaf21eb6156485f89b752897c5b8a90"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "cf4e6a95f05fe4fc587f6d84c4fc7f19"
server_key_iv: "f6157f5296f7cc5a276570ba6bb43554"
server_key_version: "encryptionkey@0"
I20260621 05:16:39.011960 9071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:39.026639 9071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:39.028304 9071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:39.028452 9071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:39.032092 9071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43773
I20260621 05:16:39.032122 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43773 every 8 connection(s)
I20260621 05:16:39.032612 9071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:16:39.033636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 9071
I20260621 05:16:39.033800 9139 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:16:39.033938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35643
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43773
--webserver_port=34137
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:16:39.034533 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Bootstrap starting.
I20260621 05:16:39.042853 9139 log.cc:824] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:39.044399 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:16:39.044656 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Bootstrap complete.
I20260621 05:16:39.046773 9139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260621 05:16:39.047006 9139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eaf21eb6156485f89b752897c5b8a90, State: Initialized, Role: FOLLOWER
I20260621 05:16:39.047204 9139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260621 05:16:39.047288 9139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:16:39.047331 9139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:16:39.047385 9139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:16:39.048367 9139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260621 05:16:39.048456 9139 leader_election.cc:302] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eaf21eb6156485f89b752897c5b8a90; no voters:
I20260621 05:16:39.048633 9139 leader_election.cc:288] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:16:39.048663 9143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:16:39.048765 9143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [term 2 LEADER]: Becoming Leader. State: Replica: 7eaf21eb6156485f89b752897c5b8a90, State: Running, Role: LEADER
I20260621 05:16:39.048841 9143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260621 05:16:39.048870 9139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:39.049134 9143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eaf21eb6156485f89b752897c5b8a90. Latest consensus state: current_term: 2 leader_uuid: "7eaf21eb6156485f89b752897c5b8a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } } }
I20260621 05:16:39.049217 9143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:39.049626 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7eaf21eb6156485f89b752897c5b8a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eaf21eb6156485f89b752897c5b8a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } } }
I20260621 05:16:39.049705 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90 [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:39.049813 9147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:16:39.050266 9147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:16:39.050395 9147 catalog_manager.cc:1259] Loaded cluster ID: 9d245cf9d4a4409e9da9972d351be45e
I20260621 05:16:39.050446 9147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:16:39.051628 9147 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:16:39.051831 9147 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7eaf21eb6156485f89b752897c5b8a90: Loaded TSK: 0
I20260621 05:16:39.052047 9147 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:16:39.112128 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:39.112308 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:39.112329 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:39.113428 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:39.113476 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:39.113502 9141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:16:39.114898 9141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34137
--tserver_master_addrs=127.0.98.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:39.115111 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:39.115342 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:39.120316 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:39.120368 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:39.120364 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:39.120617 9141 server_base.cc:1034] running on GCE node
I20260621 05:16:39.120798 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:39.121057 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:39.122207 9141 hybrid_clock.cc:648] HybridClock initialized: now 1782018999122199 us; error 25 us; skew 500 ppm
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:39.124716 9141 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260621 05:16:39.125200 9141 webserver.cc:460] Webserver started at http://127.0.98.1:34137/ using document root <none> and password file <none>
I20260621 05:16:39.125454 9141 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:39.125527 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:39.127063 9141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:16:39.127696 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.127902 9141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:39.127991 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "2e04701f4100474fbd0c7e656aaf126f"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "af7e0d1b2747544acdcf0ed9e5d398c8"
server_key_iv: "0986738a3594ee3bd72e6b05a62efdba"
server_key_version: "encryptionkey@0"
I20260621 05:16:39.128413 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:39.146227 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:39.148510 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:39.148663 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:39.148909 9141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:39.149250 9141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:39.149296 9141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.149326 9141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:39.149344 9141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.156674 9141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35643
I20260621 05:16:39.156708 9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35643 every 8 connection(s)
I20260621 05:16:39.157095 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:16:39.158955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 9141
I20260621 05:16:39.159251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34591
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43773
--webserver_port=33755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:39.167330 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.157813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:34127)
0621 05:16:39.158007 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.158051 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.158612 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.159119 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.159123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.159854 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.160113 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.160119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.161336 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.161337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.162664 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.162667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.163552 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.163553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.163743 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.164306 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.164317 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.167094 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
W20260621 05:16:39.167409 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.157923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34127 (local address 127.0.98.62:43773)
0621 05:16:39.158112 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.158118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.158141 (+ 23us) server_negotiation.cc:407] Connection header received
0621 05:16:39.158807 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.158814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.158890 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.159048 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.160245 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.161229 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.162838 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.163451 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.163569 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.167230 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34127: BlockingRecv error: recv got EOF from 127.0.98.1:34127 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
W20260621 05:16:39.168040 9289 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43773 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43773: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:39.176738 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.168269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:48621)
0621 05:16:39.168343 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.168354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.168392 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.171844 (+ 3452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.171846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.172011 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.172070 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.172072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.172975 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.172976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.173772 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.173774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.174231 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.174232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.174294 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.174736 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.174748 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.176670 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 05:16:39.176818 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.168318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48621 (local address 127.0.98.62:43773)
0621 05:16:39.171719 (+ 3401us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.171721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.171731 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:16:39.171751 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.171752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.171763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.171822 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.172153 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.172887 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.173858 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.174156 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.174223 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.176762 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48621: BlockingRecv error: recv got EOF from 127.0.98.1:48621 (error 108)
Metrics: {"server-negotiator.queue_time_us":3379}
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:39.181831 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.177184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:33623)
0621 05:16:39.177223 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.177233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.177262 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.177378 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.177379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.177503 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.177531 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.177532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.178421 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.178422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.179037 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.179039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.179478 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.179478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.179523 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.179983 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.179988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.181791 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:16:39.181937 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.177238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33623 (local address 127.0.98.62:43773)
0621 05:16:39.177267 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.177269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.177278 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:39.177300 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.177300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.177309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.177365 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.177598 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.178333 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.179117 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.179400 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.179476 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.181890 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33623: BlockingRecv error: recv got EOF from 127.0.98.1:33623 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260621 05:16:39.182025 9289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:16:39.242724 9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:39.242914 9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:39.242937 9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:39.244017 9293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:39.244050 9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:39.244073 9293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:39.245402 9293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33755
--tserver_master_addrs=127.0.98.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:39.245570 9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:39.245729 9293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:39.250555 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:39.250532 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:39.250559 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:39.250712 9293 server_base.cc:1034] running on GCE node
I20260621 05:16:39.251003 9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:39.251214 9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:39.252343 9293 hybrid_clock.cc:648] HybridClock initialized: now 1782018999252331 us; error 28 us; skew 500 ppm
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:39.254693 9293 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260621 05:16:39.255167 9293 webserver.cc:460] Webserver started at http://127.0.98.2:33755/ using document root <none> and password file <none>
I20260621 05:16:39.255376 9293 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:39.255431 9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:39.256735 9293 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:39.257257 9308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.257416 9293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:39.257504 9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "bb0357d93a62424bb7c393467a78a974"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "8c8df475a5747ab6b719cc2186b03aac"
server_key_iv: "4a574acb76e7385cef5790076f370ea5"
server_key_version: "encryptionkey@0"
I20260621 05:16:39.257790 9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:39.265164 9293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:39.267167 9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:39.267306 9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:39.267499 9293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:39.267823 9293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:39.267874 9293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.267912 9293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:39.267938 9293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.273576 9293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34591
I20260621 05:16:39.273615 9421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34591 every 8 connection(s)
I20260621 05:16:39.274005 9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:16:39.274487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 9293
I20260621 05:16:39.274821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34123
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43773
--webserver_port=44811
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:39.281198 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.274616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:53679)
0621 05:16:39.274769 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.274807 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.275277 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.275514 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.275518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.276273 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.276522 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.276528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.277135 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.277136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.277905 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.277907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.278267 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.278268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.278404 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.278915 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.278924 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.281069 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
W20260621 05:16:39.281263 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.274653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53679 (local address 127.0.98.62:43773)
0621 05:16:39.274708 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.274711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.274899 (+ 188us) server_negotiation.cc:407] Connection header received
0621 05:16:39.275384 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.275385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.275401 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.275454 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.276620 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.277057 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.278009 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.278182 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.278259 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.281180 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53679: BlockingRecv error: recv got EOF from 127.0.98.2:53679 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260621 05:16:39.281455 9422 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43773 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43773: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:39.285676 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.281684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:34789)
0621 05:16:39.281766 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.281777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.281838 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.281967 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.281968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.282077 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.282112 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.282113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.282664 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.282666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.283120 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.283122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.283448 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.283449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.283509 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.283918 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.283924 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.285603 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260621 05:16:39.285724 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.281709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34789 (local address 127.0.98.62:43773)
0621 05:16:39.281743 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.281745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.281827 (+ 82us) server_negotiation.cc:407] Connection header received
0621 05:16:39.281904 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.281904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.281913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.281961 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.282186 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.282593 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.283207 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.283372 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.283440 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.285665 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34789: BlockingRecv error: recv got EOF from 127.0.98.2:34789 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:39.289494 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.285888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:42163)
0621 05:16:39.285986 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.285993 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.286032 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.286140 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.286140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.286250 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.286269 (+ 19us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.286270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.286810 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.286811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.287234 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.287236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.287557 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.287557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.287602 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.287990 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.287994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.289423 (+ 1429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
W20260621 05:16:39.289515 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.285913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42163 (local address 127.0.98.62:43773)
0621 05:16:39.285944 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.285946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.286043 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:16:39.286080 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.286080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.286088 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.286136 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.286337 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.286732 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.287306 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.287475 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.287555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.289482 (+ 1927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42163: BlockingRecv error: recv got EOF from 127.0.98.2:42163 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260621 05:16:39.289615 9422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:16:39.351583 9425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:39.351787 9425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:39.351809 9425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:39.352883 9425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:39.352917 9425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:39.352943 9425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:39.354274 9425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44811
--tserver_master_addrs=127.0.98.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:39.354444 9425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:39.354580 9425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:39.359109 9432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:39.359115 9431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:39.359122 9434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:39.359431 9425 server_base.cc:1034] running on GCE node
I20260621 05:16:39.359651 9425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:39.359889 9425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:39.361019 9425 hybrid_clock.cc:648] HybridClock initialized: now 1782018999361009 us; error 25 us; skew 500 ppm
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:39.363281 9425 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260621 05:16:39.363709 9425 webserver.cc:460] Webserver started at http://127.0.98.3:44811/ using document root <none> and password file <none>
I20260621 05:16:39.363911 9425 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:39.363986 9425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:39.365216 9425 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:39.365769 9441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.365945 9425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:39.366051 9425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "fa805e8a26bb4549b5ca8501a815615e"
format_stamp: "Formatted at 2026-06-21 05:16:38 on dist-test-slave-czm5"
server_key: "99f772ce3de38db4049a2b75bf827266"
server_key_iv: "234d8f13b4515dd0c544f1d6af411060"
server_key_version: "encryptionkey@0"
I20260621 05:16:39.366343 9425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:39.399848 9425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:39.401535 9425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:39.401702 9425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:39.401901 9425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:39.402184 9425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:39.402233 9425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.402266 9425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:39.402289 9425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:39.407526 9425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34123
I20260621 05:16:39.407599 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34123 every 8 connection(s)
I20260621 05:16:39.407949 9425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:16:39.409806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 9425
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:16:39.415320 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.408615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43807 (local address 127.0.98.62:43773)
0621 05:16:39.408659 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.408661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.408897 (+ 236us) server_negotiation.cc:407] Connection header received
0621 05:16:39.409270 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.409270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.409279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.409325 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.410335 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.410827 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.412078 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.412234 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.412279 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.415258 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43807: BlockingRecv error: recv got EOF from 127.0.98.3:43807 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:16:39.415347 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.408620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:43807)
0621 05:16:39.408780 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.408822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.409200 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.409371 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.409373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.410060 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.410228 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.410233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.410911 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.410912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.411984 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.411986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.412304 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.412304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.412424 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.412827 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.412836 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.415166 (+ 2330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260621 05:16:39.415571 9555 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43773 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43773: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260621 05:16:39.417562 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.410322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.0.98.62:43773)
0621 05:16:39.410421 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.410423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.410560 (+ 137us) server_negotiation.cc:407] Connection header received
0621 05:16:39.410631 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.410632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.410643 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.410718 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.411469 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.411965 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.412896 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.413059 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.413114 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.414431 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:39.414435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:39.414435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:39.414444 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:39.416300 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:39.416637 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:39.416640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:39.416641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:39.416734 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:39.416947 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:39.416947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:39.416948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:39.417144 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:39.417273 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:39.417369 (+ 96us) server_negotiation.cc:299] Negotiation successful
0621 05:16:39.417474 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":57,"threads_started":1}
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:39.419535 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.415789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:43921)
0621 05:16:39.415839 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.415849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.415886 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.416027 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.416028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.416148 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.416184 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.416185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.416789 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.416790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.417250 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.417251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.417607 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.417608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.417662 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.418047 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.418051 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.419501 (+ 1450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:16:39.419656 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.415848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43921 (local address 127.0.98.62:43773)
0621 05:16:39.415888 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.415890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.415898 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:39.415961 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.415962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.415970 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.416022 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.416280 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.416703 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.417324 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.417550 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.417598 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.419589 (+ 1991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43921: BlockingRecv error: recv got EOF from 127.0.98.3:43921 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:39 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:39.423038 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:39.419744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:58191)
0621 05:16:39.419803 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:39.419811 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:39.419840 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:39.419938 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:39.419938 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:39.420024 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:39.420044 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.420044 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.420561 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.420561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.421005 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:39.421006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.421285 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.421285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:39.421316 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.421578 (+ 262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:39.421582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:39.423001 (+ 1419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 05:16:39.423153 9555 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:16:39.423154 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:39.419812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58191 (local address 127.0.98.62:43773)
0621 05:16:39.419855 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:39.419857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:39.419866 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:39.419885 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:39.419889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:39.419895 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:39.419936 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:39.420080 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.420485 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.421068 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:39.421227 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:39.421283 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:39.423089 (+ 1806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58191: BlockingRecv error: recv got EOF from 127.0.98.3:58191 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 21 05:16:40 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:40 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:40.186430 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:40.182349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:54179)
0621 05:16:40.182484 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:40.182493 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:40.182536 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:40.182667 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:40.182667 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:40.182807 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:40.182851 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:40.182853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.183477 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.183478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:40.184003 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:40.184005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.184348 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.184349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:40.184384 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:40.184784 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:40.184788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:40.186365 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
W20260621 05:16:40.186517 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:40.182392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54179 (local address 127.0.98.62:43773)
0621 05:16:40.182516 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:40.182517 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:40.182525 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:40.182588 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:40.182588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:40.182597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:40.182653 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:40.182933 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.183414 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.184072 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.184281 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.184337 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:40.186448 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54179: BlockingRecv error: recv got EOF from 127.0.98.1:54179 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
Jun 21 05:16:40 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:40 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:40.294401 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:40.289936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:36949)
0621 05:16:40.290080 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:40.290092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:40.290151 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:40.290299 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:40.290299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:40.290487 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:40.290534 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:40.290535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.291220 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.291221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:40.291786 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:40.291788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.292142 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.292143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:40.292205 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:40.292628 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:40.292632 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:40.294338 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
W20260621 05:16:40.294543 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:40.290004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36949 (local address 127.0.98.62:43773)
0621 05:16:40.290072 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:40.290073 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:40.290140 (+ 67us) server_negotiation.cc:407] Connection header received
0621 05:16:40.290200 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:40.290201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:40.290212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:40.290287 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:40.290644 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.291136 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.291874 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.292061 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.292134 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:40.294443 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36949: BlockingRecv error: recv got EOF from 127.0.98.2:36949 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 05:16:40 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:40 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:40.427994 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:40.423514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:46333)
0621 05:16:40.423647 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:40.423675 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:40.423737 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:40.423875 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:40.423875 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:40.424065 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:40.424144 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:40.424147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.424927 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.424928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:40.425461 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:40.425462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.425774 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.425775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:40.425818 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:40.426232 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:40.426236 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:40.427897 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
W20260621 05:16:40.428026 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:40.423552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46333 (local address 127.0.98.62:43773)
0621 05:16:40.423595 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:40.423597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:40.423743 (+ 146us) server_negotiation.cc:407] Connection header received
0621 05:16:40.423788 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:40.423788 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:40.423804 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:40.423869 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:40.424211 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.424849 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.425534 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:40.425711 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:40.425764 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:40.427965 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46333: BlockingRecv error: recv got EOF from 127.0.98.3:46333 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 21 05:16:41 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:41 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:41.191354 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:41.187160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:36387)
0621 05:16:41.187268 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:41.187285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:41.187331 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:41.187475 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:41.187476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:41.187645 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:41.187691 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:41.187692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.188288 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.188289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:41.188785 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:41.188786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.189085 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.189086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:41.189121 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:41.189617 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:41.189622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:41.191290 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
W20260621 05:16:41.191448 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:41.187217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36387 (local address 127.0.98.62:43773)
0621 05:16:41.187350 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:41.187351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:41.187359 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:41.187378 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:41.187379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:41.187388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:41.187456 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:41.187756 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.188224 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.188857 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.189026 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.189073 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:41.191374 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36387: BlockingRecv error: recv got EOF from 127.0.98.1:36387 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
Jun 21 05:16:41 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:41 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:41.299407 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:41.295045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:49111)
0621 05:16:41.295172 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:41.295183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:41.295241 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:41.295371 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:41.295372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:41.295528 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:41.295575 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:41.295577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.296192 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.296193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:41.296874 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:41.296877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.297227 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.297228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:41.297280 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:41.297687 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:41.297691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:41.299276 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
W20260621 05:16:41.299435 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:41.295131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49111 (local address 127.0.98.62:43773)
0621 05:16:41.295201 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:41.295202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:41.295230 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:16:41.295292 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:41.295292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:41.295302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:41.295353 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:41.295654 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.296128 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.296948 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.297147 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.297228 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:41.299369 (+ 2141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49111: BlockingRecv error: recv got EOF from 127.0.98.2:49111 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 21 05:16:41 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:41 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:41.432577 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:41.428427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:48221)
0621 05:16:41.428547 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:41.428559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:41.428611 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:41.428747 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:41.428748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:41.428895 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:41.428941 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:41.428942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.429526 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.429526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:41.430070 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:41.430071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.430363 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.430363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:41.430400 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:41.430806 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:41.430811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:41.432520 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"spinlock_wait_cycles":16128,"thread_start_us":98,"threads_started":1}
W20260621 05:16:41.432699 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:41.428539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48221 (local address 127.0.98.62:43773)
0621 05:16:41.428607 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:41.428609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:41.428616 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:16:41.428672 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:41.428672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:41.428682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:41.428728 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:41.429000 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.429466 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.430137 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:41.430304 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:41.430352 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:41.432602 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48221: BlockingRecv error: recv got EOF from 127.0.98.3:48221 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 05:16:42 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:42 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:42.195935 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:42.191790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:43175)
0621 05:16:42.191939 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:42.191949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:42.191995 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:42.192140 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:42.192141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:42.192282 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:42.192328 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:42.192329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.192914 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.192914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:42.193378 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:42.193379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.193676 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.193677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:42.193713 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:42.194177 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:42.194182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:42.195880 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
W20260621 05:16:42.196063 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:42.191881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43175 (local address 127.0.98.62:43773)
0621 05:16:42.192005 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:42.192023 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:42.192033 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:16:42.192060 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:42.192061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:42.192070 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:42.192124 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:42.192386 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.192852 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.193435 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.193603 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.193666 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:42.195966 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43175: BlockingRecv error: recv got EOF from 127.0.98.1:43175 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
Jun 21 05:16:42 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:42 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:42.304800 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:42.300068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:49471)
0621 05:16:42.300192 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:42.300207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:42.300252 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:42.300435 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:42.300436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:42.300607 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:42.300651 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:42.300653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.301336 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.301337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:42.301851 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:42.301853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.302345 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.302346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:42.302425 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:42.303023 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:42.303028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:42.304750 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
W20260621 05:16:42.304901 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:42.300236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49471 (local address 127.0.98.62:43773)
0621 05:16:42.300317 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:42.300319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:42.300326 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:16:42.300356 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:42.300356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:42.300365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:42.300428 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:42.300744 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.301256 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.301941 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.302149 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.302232 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:42.304817 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49471: BlockingRecv error: recv got EOF from 127.0.98.2:49471 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 05:16:42 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:42 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:42.437209 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:42.432957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:59745)
0621 05:16:42.433083 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:42.433093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:42.433153 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:42.433267 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:42.433267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:42.433409 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:42.433453 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:42.433455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.434039 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.434039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:42.434498 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:42.434499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.434803 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.434804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:42.434859 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:42.435333 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:42.435339 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:42.437157 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
W20260621 05:16:42.437300 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:42.433014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59745 (local address 127.0.98.62:43773)
0621 05:16:42.433081 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:42.433083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:42.433151 (+ 68us) server_negotiation.cc:407] Connection header received
0621 05:16:42.433192 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:42.433192 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:42.433204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:42.433254 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:42.433536 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.433979 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.434559 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:42.434738 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:42.434795 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:42.437238 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59745: BlockingRecv error: recv got EOF from 127.0.98.3:59745 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 21 05:16:43 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:43 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:43.200480 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:43.196424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:38425)
0621 05:16:43.196573 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:43.196582 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:43.196630 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:43.196744 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:43.196744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:43.196884 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:43.196926 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:43.196928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.197490 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.197491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:43.197949 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:43.197950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.198264 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.198265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:43.198298 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:43.198700 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:43.198704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:43.200425 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
W20260621 05:16:43.200613 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:43.196501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38425 (local address 127.0.98.62:43773)
0621 05:16:43.196616 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:43.196618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:43.196626 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:43.196679 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:43.196680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:43.196689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:43.196729 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:43.196983 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.197430 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.198040 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.198207 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.198253 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:43.200526 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38425: BlockingRecv error: recv got EOF from 127.0.98.1:38425 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
Jun 21 05:16:43 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:43 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:43.309818 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:43.305364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:43749)
0621 05:16:43.305543 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:43.305555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:43.305602 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:43.305715 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:43.305716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:43.305877 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:43.305928 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:43.305930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.306547 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.306549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:43.307028 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:43.307030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.307372 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.307373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:43.307424 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:43.307991 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:43.307995 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:43.309734 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
W20260621 05:16:43.309865 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:43.305497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43749 (local address 127.0.98.62:43773)
0621 05:16:43.305578 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:43.305580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:43.305602 (+ 22us) server_negotiation.cc:407] Connection header received
0621 05:16:43.305656 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:43.305657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:43.305668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:43.305711 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:43.306005 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.306467 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.307108 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.307305 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.307374 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:43.309807 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43749: BlockingRecv error: recv got EOF from 127.0.98.2:43749 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 05:16:43 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:43 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:43.445345 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:43.437625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:46653)
0621 05:16:43.437763 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:43.437773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:43.437831 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:43.437956 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:43.437957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:43.438100 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:43.438144 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:43.438146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.438732 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.438732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:43.439225 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:43.439227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.439512 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.439512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:43.439547 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:43.443451 (+ 3904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:43.443455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:43.445290 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
W20260621 05:16:43.445453 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:43.437690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46653 (local address 127.0.98.62:43773)
0621 05:16:43.437773 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:43.437775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:43.437816 (+ 41us) server_negotiation.cc:407] Connection header received
0621 05:16:43.437876 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:43.437876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:43.437887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:43.437932 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:43.438205 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.438674 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.439291 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:43.439457 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:43.439503 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:43.445378 (+ 5875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46653: BlockingRecv error: recv got EOF from 127.0.98.3:46653 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 21 05:16:44 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:44 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:44.205368 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:44.200991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:52365)
0621 05:16:44.201182 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:44.201194 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:44.201260 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:44.201399 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:44.201400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:44.201559 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:44.201607 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:44.201608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.202231 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.202232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:44.202788 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:44.202790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.203099 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.203100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:44.203137 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:44.203586 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:44.203590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:44.205299 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":106,"threads_started":1}
W20260621 05:16:44.205499 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:44.201143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52365 (local address 127.0.98.62:43773)
0621 05:16:44.201273 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:44.201275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:44.201283 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:44.201315 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:44.201316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:44.201326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:44.201376 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:44.201673 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.202164 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.202857 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.203039 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.203086 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:44.205392 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52365: BlockingRecv error: recv got EOF from 127.0.98.1:52365 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
Jun 21 05:16:44 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:44 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:44.315021 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:44.310351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:54261)
0621 05:16:44.310498 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:44.310512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:44.310559 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:44.310685 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:44.310686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:44.310854 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:44.310900 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:44.310901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.311547 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.311548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:44.312495 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:44.312497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.312958 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.312959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:44.313008 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:44.313382 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:44.313386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:44.314968 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
W20260621 05:16:44.315131 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:44.310510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54261 (local address 127.0.98.62:43773)
0621 05:16:44.310569 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:44.310571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:44.310597 (+ 26us) server_negotiation.cc:407] Connection header received
0621 05:16:44.310618 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:44.310618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:44.310628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:44.310664 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:44.310979 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.311475 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.312582 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.312876 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.312942 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:44.315060 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54261: BlockingRecv error: recv got EOF from 127.0.98.2:54261 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 21 05:16:44 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:44 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:44.450745 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:44.445799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:50569)
0621 05:16:44.445959 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:44.445971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:44.446061 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:44.446201 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:44.446201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:44.446370 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:44.446413 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:44.446415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.447208 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.447210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:44.447824 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:44.447826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.448182 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.448183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:44.448239 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:44.448800 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:44.448827 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:44.450646 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
W20260621 05:16:44.450773 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:44.445884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50569 (local address 127.0.98.62:43773)
0621 05:16:44.445956 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:44.445958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:44.446041 (+ 83us) server_negotiation.cc:407] Connection header received
0621 05:16:44.446132 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:44.446133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:44.446147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:44.446197 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:44.446513 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.447129 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.447911 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:44.448102 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:44.448181 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:44.450716 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50569: BlockingRecv error: recv got EOF from 127.0.98.3:50569 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 21 05:16:45 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:45 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:45.210402 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:45.205929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:41787)
0621 05:16:45.206095 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:45.206105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:45.206157 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:45.206305 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:45.206305 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:45.206512 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:45.206565 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:45.206567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.207385 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.207387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:45.207930 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:45.207931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.208240 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.208241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:45.208277 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:45.208702 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:45.208706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:45.210349 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
W20260621 05:16:45.210533 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:45.206009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41787 (local address 127.0.98.62:43773)
0621 05:16:45.206137 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:45.206138 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:45.206147 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:45.206215 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:45.206216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:45.206226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:45.206283 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:45.206665 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.207320 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.207996 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.208180 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.208230 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:45.210447 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41787: BlockingRecv error: recv got EOF from 127.0.98.1:41787 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":51,"threads_started":1}
Jun 21 05:16:45 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:45 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:45.319506 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:45.315455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:60507)
0621 05:16:45.315587 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:45.315598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:45.315676 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:45.315797 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:45.315798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:45.315951 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:45.315999 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:45.316001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.316599 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.316599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:45.317123 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:45.317124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.317407 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.317408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:45.317443 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:45.317835 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:45.317839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:45.319443 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
W20260621 05:16:45.319609 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:45.315530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60507 (local address 127.0.98.62:43773)
0621 05:16:45.315574 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:45.315577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:45.315665 (+ 88us) server_negotiation.cc:407] Connection header received
0621 05:16:45.315721 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:45.315721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:45.315730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:45.315780 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:45.316073 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.316539 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.317187 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.317351 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.317397 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:45.319525 (+ 2128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60507: BlockingRecv error: recv got EOF from 127.0.98.2:60507 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 21 05:16:45 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:45 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:45.455987 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:45.451176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:40771)
0621 05:16:45.451335 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:45.451347 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:45.451434 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:45.451608 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:45.451609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:45.451803 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:45.451848 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:45.451850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.452511 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.452513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:45.453013 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:45.453014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.453363 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.453364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:45.453425 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:45.453866 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:45.453870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:45.455881 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
W20260621 05:16:45.456022 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:45.451255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40771 (local address 127.0.98.62:43773)
0621 05:16:45.451325 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:45.451328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:45.451416 (+ 88us) server_negotiation.cc:407] Connection header received
0621 05:16:45.451517 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:45.451518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:45.451535 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:45.451601 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:45.451930 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.452429 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.453105 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:45.453293 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:45.453365 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:45.455956 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40771: BlockingRecv error: recv got EOF from 127.0.98.3:40771 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 21 05:16:46 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:46 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:46.215287 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:46.210907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:54387)
0621 05:16:46.211064 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:46.211073 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:46.211123 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:46.211271 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:46.211271 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:46.211418 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:46.211464 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:46.211466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.212116 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.212117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:46.212645 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:46.212647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.212960 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.212961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:46.212996 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:46.213419 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:46.213424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:46.215184 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
W20260621 05:16:46.215308 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:46.210965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54387 (local address 127.0.98.62:43773)
0621 05:16:46.211100 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:46.211102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:46.211111 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:46.211182 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:46.211183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:46.211193 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:46.211252 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:46.211524 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.212052 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.212712 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.212899 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.212950 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:46.215260 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54387: BlockingRecv error: recv got EOF from 127.0.98.1:54387 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
Jun 21 05:16:46 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:46 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:46.324827 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:46.319974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:58509)
0621 05:16:46.320147 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:46.320164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:46.320234 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:46.320384 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:46.320384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:46.320546 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:46.320587 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:46.320588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.321241 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.321242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:46.321784 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:46.321786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.322151 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.322152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:46.322211 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:46.322703 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:46.322708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:46.324741 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
W20260621 05:16:46.324889 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:46.320051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58509 (local address 127.0.98.62:43773)
0621 05:16:46.320117 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:46.320119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:46.320216 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:16:46.320294 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:46.320295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:46.320305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:46.320379 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:46.320657 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.321131 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.321881 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.322070 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.322151 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:46.324807 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58509: BlockingRecv error: recv got EOF from 127.0.98.2:58509 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 05:16:46 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:46 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:46.460479 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:46.456547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:58003)
0621 05:16:46.456670 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:46.456679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:46.456730 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:46.456831 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:46.456832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:46.456969 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:46.457012 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:46.457013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.457561 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.457561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:46.458036 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:46.458044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.458320 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.458320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:46.458352 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:46.458722 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:46.458726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:46.460368 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
W20260621 05:16:46.460585 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:46.456590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58003 (local address 127.0.98.62:43773)
0621 05:16:46.456647 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:46.456649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:46.456729 (+ 80us) server_negotiation.cc:407] Connection header received
0621 05:16:46.456767 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:46.456768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:46.456780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:46.456827 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:46.457060 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.457504 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.458115 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:46.458268 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:46.458310 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:46.460506 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58003: BlockingRecv error: recv got EOF from 127.0.98.3:58003 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 21 05:16:47 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:47 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:47.219873 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:47.215754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:33017)
0621 05:16:47.215867 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:47.215877 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:47.215932 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:47.216087 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:47.216088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:47.216224 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:47.216269 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:47.216270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.216820 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.216821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:47.217287 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:47.217288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.217562 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.217563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:47.217599 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:47.218139 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:47.218148 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:47.219787 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
W20260621 05:16:47.219919 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:47.215848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33017 (local address 127.0.98.62:43773)
0621 05:16:47.215960 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:47.215962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:47.215969 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:16:47.215997 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:47.215997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:47.216007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:47.216081 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:47.216323 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.216764 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.217345 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.217506 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.217551 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:47.219854 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33017: BlockingRecv error: recv got EOF from 127.0.98.1:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
Jun 21 05:16:47 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:47 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:47.329421 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:47.325375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:46495)
0621 05:16:47.325474 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:47.325484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:47.325542 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:47.325653 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:47.325654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:47.325811 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:47.325856 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:47.325858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.326551 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.326552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:47.327127 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:47.327128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.327422 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.327423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:47.327460 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:47.327886 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:47.327890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:47.329370 (+ 1480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
W20260621 05:16:47.329509 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:47.325435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46495 (local address 127.0.98.62:43773)
0621 05:16:47.325508 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:47.325509 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:47.325530 (+ 21us) server_negotiation.cc:407] Connection header received
0621 05:16:47.325584 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:47.325585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:47.325594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:47.325638 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:47.325919 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.326494 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.327196 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.327365 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.327411 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:47.329449 (+ 2038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46495: BlockingRecv error: recv got EOF from 127.0.98.2:46495 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 21 05:16:47 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:47 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:47.465458 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:47.460945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:44235)
0621 05:16:47.461070 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:47.461080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:47.461136 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:47.461379 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:47.461380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:47.461579 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:47.461626 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:47.461627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.462331 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.462332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:47.462841 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:47.462843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.463167 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.463168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:47.463219 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:47.463712 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:47.463718 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:47.465404 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
W20260621 05:16:47.465593 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:47.461008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44235 (local address 127.0.98.62:43773)
0621 05:16:47.461080 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:47.461082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:47.461144 (+ 62us) server_negotiation.cc:407] Connection header received
0621 05:16:47.461251 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:47.461251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:47.461284 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:47.461364 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:47.461727 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.462266 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.462910 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:47.463110 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:47.463157 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:47.465488 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44235: BlockingRecv error: recv got EOF from 127.0.98.3:44235 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 21 05:16:48 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:48 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:48.224911 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:48.220331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:42671)
0621 05:16:48.220485 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:48.220495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:48.220548 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:48.220710 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:48.220711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:48.220871 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:48.220920 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:48.220922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.221548 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.221548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:48.222063 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:48.222064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.222426 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.222427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:48.222497 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:48.223026 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:48.223032 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:48.224818 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
W20260621 05:16:48.224956 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:48.220409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42671 (local address 127.0.98.62:43773)
0621 05:16:48.220519 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:48.220521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:48.220533 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:16:48.220613 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:48.220614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:48.220624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:48.220681 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:48.220985 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.221484 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.222139 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.222346 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.222426 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:48.224890 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42671: BlockingRecv error: recv got EOF from 127.0.98.1:42671 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
Jun 21 05:16:48 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:48 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:48.334489 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:48.329875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:46813)
0621 05:16:48.330038 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:48.330055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:48.330137 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:48.330276 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:48.330277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:48.330526 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:48.330583 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:48.330585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.331196 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.331197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:48.331805 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:48.331807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.332144 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.332145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:48.332186 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:48.332631 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:48.332635 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:48.334431 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
W20260621 05:16:48.334587 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:48.329979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46813 (local address 127.0.98.62:43773)
0621 05:16:48.330057 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:48.330059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:48.330142 (+ 83us) server_negotiation.cc:407] Connection header received
0621 05:16:48.330185 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:48.330186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:48.330199 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:48.330258 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:48.330657 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.331104 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.331891 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.332081 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.332133 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:48.334519 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46813: BlockingRecv error: recv got EOF from 127.0.98.2:46813 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 21 05:16:48 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:48 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:48.474723 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:48.465980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:36227)
0621 05:16:48.466158 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:48.466175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:48.466252 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:48.466434 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:48.466435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:48.466603 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:48.466666 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:48.466668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.467313 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.467314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:48.467940 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:48.467943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.468408 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.468409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:48.468512 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:48.472962 (+ 4450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:48.472967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:48.474666 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":109,"threads_started":1}
W20260621 05:16:48.474789 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:48.466069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36227 (local address 127.0.98.62:43773)
0621 05:16:48.466141 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:48.466143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:48.466233 (+ 90us) server_negotiation.cc:407] Connection header received
0621 05:16:48.466346 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:48.466346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:48.466365 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:48.466428 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:48.466752 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.467229 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.468025 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:48.468297 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:48.468392 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:48.474744 (+ 6352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36227: BlockingRecv error: recv got EOF from 127.0.98.3:36227 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 21 05:16:49 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:49 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:49.229827 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:49.225512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:48639)
0621 05:16:49.225651 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:49.225668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:49.225714 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:49.225888 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:49.225889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:49.226042 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:49.226090 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:49.226091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.226737 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.226738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:49.227283 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:49.227284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.227598 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.227598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:49.227699 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:49.228159 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:49.228163 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:49.229770 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260621 05:16:49.229938 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:49.225609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48639 (local address 127.0.98.62:43773)
0621 05:16:49.225730 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:49.225736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:49.225744 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:49.225763 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:49.225764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:49.225773 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:49.225835 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:49.226159 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.226658 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.227355 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.227537 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.227585 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:49.229860 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48639: BlockingRecv error: recv got EOF from 127.0.98.1:48639 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
Jun 21 05:16:49 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:49 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:49.339350 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:49.334937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:42121)
0621 05:16:49.335105 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:49.335117 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:49.335204 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:49.335351 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:49.335352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:49.335521 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:49.335566 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:49.335569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.336261 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.336263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:49.336736 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:49.336738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.337079 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.337080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:49.337140 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:49.337601 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:49.337605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:49.339258 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
W20260621 05:16:49.339411 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:49.335031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42121 (local address 127.0.98.62:43773)
0621 05:16:49.335102 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:49.335105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:49.335187 (+ 82us) server_negotiation.cc:407] Connection header received
0621 05:16:49.335279 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:49.335280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:49.335294 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:49.335346 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:49.335660 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.336185 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.336824 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.337009 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.337076 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:49.339342 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42121: BlockingRecv error: recv got EOF from 127.0.98.2:42121 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 21 05:16:49 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:49 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:49.479333 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:49.475250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:57177)
0621 05:16:49.475403 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:49.475415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:49.475484 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:49.475605 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:49.475605 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:49.475806 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:49.475853 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:49.475854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.476445 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.476446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:49.476908 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:49.476909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.477207 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.477208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:49.477245 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:49.477642 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:49.477646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:49.479265 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
W20260621 05:16:49.479660 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:49.475380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57177 (local address 127.0.98.62:43773)
0621 05:16:49.475442 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:49.475444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:49.475468 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:16:49.475525 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:49.475526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:49.475537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:49.475592 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:49.475917 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.476388 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.476972 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:49.477151 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:49.477196 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:49.479374 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57177: BlockingRecv error: recv got EOF from 127.0.98.3:57177 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 21 05:16:50 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:50 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:50.234833 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:50.230326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:43991)
0621 05:16:50.230500 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:50.230510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:50.230559 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:50.230668 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:50.230669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:50.230810 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:50.230855 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:50.230857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.231457 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.231458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:50.232009 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:50.232010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.232403 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.232404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:50.232446 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:50.232896 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:50.232901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:50.234746 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
W20260621 05:16:50.234880 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:50.230417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43991 (local address 127.0.98.62:43773)
0621 05:16:50.230577 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:50.230578 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:50.230587 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:50.230605 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:50.230605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:50.230614 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:50.230664 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:50.230909 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.231396 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.232072 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.232340 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.232392 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:50.234804 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43991: BlockingRecv error: recv got EOF from 127.0.98.1:43991 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":112,"threads_started":1}
Jun 21 05:16:50 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:50 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:50.344259 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:50.339837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:55469)
0621 05:16:50.339998 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:50.340007 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:50.340059 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:50.340200 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:50.340201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:50.340348 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:50.340395 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:50.340397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.340992 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.340993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:50.341814 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:50.341818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.342134 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.342135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:50.342173 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:50.342582 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:50.342587 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:50.344205 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
W20260621 05:16:50.344349 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:50.339916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55469 (local address 127.0.98.62:43773)
0621 05:16:50.339981 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:50.339983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:50.340061 (+ 78us) server_negotiation.cc:407] Connection header received
0621 05:16:50.340121 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:50.340122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:50.340132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:50.340183 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:50.340453 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.340932 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.341877 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.342072 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.342126 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:50.344287 (+ 2161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55469: BlockingRecv error: recv got EOF from 127.0.98.2:55469 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 21 05:16:50 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:50 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:50.484131 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:50.480012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:33329)
0621 05:16:50.480166 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:50.480178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:50.480246 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:50.480393 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:50.480393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:50.480589 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:50.480637 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:50.480638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.481251 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.481252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:50.481776 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:50.481777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.482071 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.482072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:50.482109 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:50.482524 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:50.482528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:50.484077 (+ 1549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
W20260621 05:16:50.484248 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:50.480104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33329 (local address 127.0.98.62:43773)
0621 05:16:50.480153 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:50.480155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:50.480249 (+ 94us) server_negotiation.cc:407] Connection header received
0621 05:16:50.480301 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:50.480301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:50.480315 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:50.480372 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:50.480704 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.481188 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.481845 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:50.482012 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:50.482059 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:50.484157 (+ 2098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33329: BlockingRecv error: recv got EOF from 127.0.98.3:33329 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 21 05:16:51 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:51 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:51.239490 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:51.235327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:33697)
0621 05:16:51.235471 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:51.235485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:51.235548 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:51.235688 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:51.235689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:51.235845 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:51.235892 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:51.235894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.236507 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.236507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:51.237014 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:51.237015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.237336 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.237337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:51.237373 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:51.237789 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:51.237793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:51.239434 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260621 05:16:51.239595 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:51.235396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33697 (local address 127.0.98.62:43773)
0621 05:16:51.235538 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:51.235540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:51.235547 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:16:51.235592 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:51.235592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:51.235602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:51.235672 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:51.235958 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.236447 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.237082 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.237278 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.237327 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:51.239522 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33697: BlockingRecv error: recv got EOF from 127.0.98.1:33697 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
Jun 21 05:16:51 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:51 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:51.349467 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:51.344725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:45709)
0621 05:16:51.344981 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:51.344992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:51.345078 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:51.345249 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:51.345251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:51.345425 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:51.345471 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:51.345473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.346144 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.346145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:51.346691 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:51.346693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.347049 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.347050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:51.347113 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:51.347689 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:51.347693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:51.349376 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":183,"threads_started":1}
W20260621 05:16:51.349505 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:51.344893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45709 (local address 127.0.98.62:43773)
0621 05:16:51.344977 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:51.344979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:51.345058 (+ 79us) server_negotiation.cc:407] Connection header received
0621 05:16:51.345157 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:51.345158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:51.345170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:51.345245 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:51.345551 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.346056 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.346782 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.346969 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.347050 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:51.349448 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45709: BlockingRecv error: recv got EOF from 127.0.98.2:45709 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 05:16:51 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:51 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:51.489128 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:51.484544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:56823)
0621 05:16:51.484703 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:51.484716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:51.484803 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:51.484939 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:51.484940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:51.485106 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:51.485151 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:51.485153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.485838 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.485839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:51.486360 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:51.486362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.486717 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.486718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:51.486793 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:51.487319 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:51.487324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:51.489058 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":51,"threads_started":1}
W20260621 05:16:51.489229 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:51.484734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56823 (local address 127.0.98.62:43773)
0621 05:16:51.484801 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:51.484803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:51.484812 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:51.484854 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:51.484855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:51.484867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:51.484931 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:51.485237 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.485738 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.486447 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:51.486635 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:51.486716 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:51.489140 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56823: BlockingRecv error: recv got EOF from 127.0.98.3:56823 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 21 05:16:52 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:52 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:52.244201 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:52.240047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:53361)
0621 05:16:52.240181 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:52.240190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:52.240232 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:52.240359 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:52.240359 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:52.240502 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:52.240559 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:52.240560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.241121 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.241122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:52.241616 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:52.241617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.241937 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.241937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:52.241974 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:52.242390 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:52.242395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:52.244136 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
W20260621 05:16:52.244266 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:52.240100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53361 (local address 127.0.98.62:43773)
0621 05:16:52.240231 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:52.240233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:52.240240 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:16:52.240278 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:52.240278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:52.240288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:52.240342 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:52.240616 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.241062 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.241679 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.241876 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.241924 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:52.244216 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53361: BlockingRecv error: recv got EOF from 127.0.98.1:53361 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
Jun 21 05:16:52 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:52 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:52.354136 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:52.349892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:41635)
0621 05:16:52.350075 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:52.350085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:52.350143 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:52.350291 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:52.350292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:52.350466 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:52.350568 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:52.350573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.351167 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.351167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:52.351729 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:52.351730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.352048 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.352048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:52.352084 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:52.352500 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:52.352504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:52.354081 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
W20260621 05:16:52.354238 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:52.350011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41635 (local address 127.0.98.62:43773)
0621 05:16:52.350066 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:52.350068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:52.350172 (+ 104us) server_negotiation.cc:407] Connection header received
0621 05:16:52.350201 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:52.350202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:52.350214 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:52.350271 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:52.350635 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.351107 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.351813 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.351989 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.352038 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:52.354171 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41635: BlockingRecv error: recv got EOF from 127.0.98.2:41635 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 21 05:16:52 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:52 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:52.493912 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:52.489573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:52601)
0621 05:16:52.489731 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:52.489741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:52.489793 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:52.489927 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:52.489927 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:52.490079 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:52.490132 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:52.490134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.490741 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.490742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:52.491251 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:52.491253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.491558 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.491559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:52.491595 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:52.492150 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:52.492155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:52.493856 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
W20260621 05:16:52.493971 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:52.489690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52601 (local address 127.0.98.62:43773)
0621 05:16:52.489736 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:52.489738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:52.489785 (+ 47us) server_negotiation.cc:407] Connection header received
0621 05:16:52.489851 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:52.489851 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:52.489864 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:52.489906 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:52.490195 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.490679 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.491318 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:52.491490 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:52.491547 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:52.493926 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52601: BlockingRecv error: recv got EOF from 127.0.98.3:52601 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 21 05:16:53 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:53 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:53.249222 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:53.244664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:49013)
0621 05:16:53.244816 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:53.244826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:53.244878 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:53.245009 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:53.245010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:53.245201 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:53.245249 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:53.245250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.245865 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.245865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:53.246375 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:53.246376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.246688 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.246688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:53.246725 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:53.247252 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:53.247257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:53.249158 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
W20260621 05:16:53.249372 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:53.244754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49013 (local address 127.0.98.62:43773)
0621 05:16:53.244863 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:53.244865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:53.244880 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:16:53.244931 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:53.244932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:53.244941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:53.244993 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:53.245318 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.245804 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.246442 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.246630 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.246677 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:53.249237 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49013: BlockingRecv error: recv got EOF from 127.0.98.1:49013 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
Jun 21 05:16:53 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:53 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:53.358876 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:53.354613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:34951)
0621 05:16:53.354743 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:53.354755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:53.354831 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:53.355011 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:53.355012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:53.355186 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:53.355234 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:53.355235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.355918 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.355919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:53.356423 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:53.356424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.356737 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.356737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:53.356781 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:53.357197 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:53.357201 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:53.358812 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260621 05:16:53.358944 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:53.354681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34951 (local address 127.0.98.62:43773)
0621 05:16:53.354728 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:53.354730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:53.354827 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:16:53.354920 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:53.354921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:53.354935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:53.354994 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:53.355298 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.355854 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.356490 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.356671 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.356720 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:53.358900 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34951: BlockingRecv error: recv got EOF from 127.0.98.2:34951 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 21 05:16:53 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:53 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:53.505293 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:53.494599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.3:37219)
0621 05:16:53.494736 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:53.494748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:53.494805 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:53.494955 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:53.494956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:53.495099 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:53.495147 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:53.495148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.495745 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.495745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:53.496241 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:53.496242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.496519 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.496520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:53.496554 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:53.503594 (+ 7040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:53.503598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:53.505239 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
W20260621 05:16:53.505405 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:53.494705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37219 (local address 127.0.98.62:43773)
0621 05:16:53.494778 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:53.494779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:53.494790 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:16:53.494871 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:53.494871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:53.494883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:53.494933 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:53.495209 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.495686 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.496309 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:53.496464 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:53.496509 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:53.505334 (+ 8825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37219: BlockingRecv error: recv got EOF from 127.0.98.3:37219 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 05:16:54 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:54 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:54.254091 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.249750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.1:39985)
0621 05:16:54.249913 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:54.249923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:54.249998 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:54.250155 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:54.250158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:54.250306 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:54.250356 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.250358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.251039 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.251040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.251545 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.251546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.251869 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.251870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.251912 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.252348 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:54.252352 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:54.254040 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
W20260621 05:16:54.254186 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:54.249817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39985 (local address 127.0.98.62:43773)
0621 05:16:54.249961 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:54.249979 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:54.249987 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:54.250072 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:54.250073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:54.250086 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:54.250141 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:54.250414 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.250963 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.251608 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.251810 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.251858 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.254117 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39985: BlockingRecv error: recv got EOF from 127.0.98.1:39985 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":43,"threads_started":1}
Jun 21 05:16:54 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:16:54 dist-test-slave-czm5 krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:16:54.364007 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.359571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43773 (local address 127.0.98.2:53331)
0621 05:16:54.359753 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:54.359762 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:54.359821 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:54.359981 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:54.359982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:54.360132 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:54.360180 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.360181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.360868 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.360870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.361412 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.361414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.361771 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.361772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.361832 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.362270 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:54.362275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:54.363889 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43773: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":125,"threads_started":1}
W20260621 05:16:54.364060 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:54.359642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53331 (local address 127.0.98.62:43773)
0621 05:16:54.359717 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:54.359724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:54.359822 (+ 98us) server_negotiation.cc:407] Connection header received
0621 05:16:54.359902 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:54.359902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:54.359913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:54.359972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:54.360235 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.360767 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.361496 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.361689 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.361762 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.363991 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53331: BlockingRecv error: recv got EOF from 127.0.98.2:53331 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:16:54.410310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 9141
I20260621 05:16:54.414530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 9293
I20260621 05:16:54.418800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 9425
I20260621 05:16:54.425664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 9071
2026-06-21T05:16:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17300 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T05:16:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:16:54Z Disabled control of system clock
I20260621 05:16:54.451412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44825
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 05:16:54.527458 9622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:54.527666 9622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:54.527685 9622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:54.528822 9622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:54.528859 9622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:54.528872 9622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:54.528882 9622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:54.528927 9622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:54.530079 9622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:54.530257 9622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:54.530508 9622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:54.535276 9630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:54.535303 9622 server_base.cc:1034] running on GCE node
W20260621 05:16:54.535287 9628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:54.535300 9627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:54.535825 9622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:54.536101 9622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:54.537259 9622 hybrid_clock.cc:648] HybridClock initialized: now 1782019014537250 us; error 46 us; skew 500 ppm
I20260621 05:16:54.538466 9622 webserver.cc:460] Webserver started at http://127.0.98.62:35195/ using document root <none> and password file <none>
I20260621 05:16:54.538717 9622 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:54.538779 9622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:54.538914 9622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:54.539924 9622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "f6ca357e334a4c29af14b242743082be"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "a1939f0744da010d4b5ec750a6ffdd41"
server_key_iv: "724a23bbb0b747ae369693e49b121783"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.540256 9622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6ca357e334a4c29af14b242743082be"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "a1939f0744da010d4b5ec750a6ffdd41"
server_key_iv: "724a23bbb0b747ae369693e49b121783"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.541455 9622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:16:54.542158 9636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.542318 9622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:54.542413 9622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "f6ca357e334a4c29af14b242743082be"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "a1939f0744da010d4b5ec750a6ffdd41"
server_key_iv: "724a23bbb0b747ae369693e49b121783"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.542490 9622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:54.573602 9622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:54.573889 9622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:54.574023 9622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:54.577919 9622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44825
I20260621 05:16:54.577950 9688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44825 every 8 connection(s)
I20260621 05:16:54.578294 9622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:16:54.578908 9689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:54.581175 9689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be: Bootstrap starting.
I20260621 05:16:54.581707 9689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:54.581966 9689 log.cc:824] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:54.582568 9689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be: No bootstrap required, opened a new log
I20260621 05:16:54.584045 9689 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ca357e334a4c29af14b242743082be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } }
I20260621 05:16:54.584163 9689 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:54.584183 9689 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ca357e334a4c29af14b242743082be, State: Initialized, Role: FOLLOWER
I20260621 05:16:54.584300 9689 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ca357e334a4c29af14b242743082be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } }
I20260621 05:16:54.584372 9689 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:16:54.584393 9689 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:16:54.584416 9689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:54.584832 9689 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ca357e334a4c29af14b242743082be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } }
I20260621 05:16:54.584904 9689 leader_election.cc:302] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6ca357e334a4c29af14b242743082be; no voters:
I20260621 05:16:54.585007 9689 leader_election.cc:288] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:16:54.585083 9692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:54.585217 9689 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:54.585240 9692 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [term 1 LEADER]: Becoming Leader. State: Replica: f6ca357e334a4c29af14b242743082be, State: Running, Role: LEADER
I20260621 05:16:54.585522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 9622
I20260621 05:16:54.585512 9692 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ca357e334a4c29af14b242743082be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } }
I20260621 05:16:54.585599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:16:54.585803 392 external_mini_cluster.cc:1428] Setting key 8bb9b52d6ef02b276174ed7a8cd5f76b
I20260621 05:16:54.586050 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6ca357e334a4c29af14b242743082be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ca357e334a4c29af14b242743082be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } } }
I20260621 05:16:54.586141 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:54.586434 9702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:16:54.586537 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6ca357e334a4c29af14b242743082be. Latest consensus state: current_term: 1 leader_uuid: "f6ca357e334a4c29af14b242743082be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ca357e334a4c29af14b242743082be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } } }
I20260621 05:16:54.586685 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:54.586944 9702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:16:54.588450 9702 catalog_manager.cc:1347] Generated new cluster ID: 226fc932b0144702b56dab70e54de51d
I20260621 05:16:54.588500 9702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:16:54.592177 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.586422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.0.98.62:44825)
0621 05:16:54.586602 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:54.586611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:54.586888 (+ 277us) server_negotiation.cc:407] Connection header received
0621 05:16:54.586998 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:54.587005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:54.587090 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:54.587227 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:54.587425 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.588741 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.589772 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.590337 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.591078 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.591256 (+ 178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:54.591259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:54.591259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:16:54.591434 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:54.591464 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:16:54.591476 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:54.591740 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:54.591769 (+ 29us) server_negotiation.cc:299] Negotiation successful
0621 05:16:54.591834 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260621 05:16:54.596292 9702 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:16:54.596745 9702 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:16:54.599961 9702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6ca357e334a4c29af14b242743082be: Generated new TSK 0
I20260621 05:16:54.600114 9702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:16:54.602289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--builtin_ntp_servers=127.0.98.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 05:16:54.679497 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:54.679741 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:54.679766 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:54.680858 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:54.680895 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:54.680919 9713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:16:54.682199 9713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:54.682425 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:54.682718 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:16:54.688355 9713 server_base.cc:1034] running on GCE node
W20260621 05:16:54.688467 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:54.688321 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:54.688329 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:54.688778 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:54.689054 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:54.690234 9713 hybrid_clock.cc:648] HybridClock initialized: now 1782019014690226 us; error 29 us; skew 500 ppm
I20260621 05:16:54.691476 9713 webserver.cc:460] Webserver started at http://127.0.98.1:38127/ using document root <none> and password file <none>
I20260621 05:16:54.691721 9713 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:54.691773 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:54.691881 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:54.693154 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "194053f0783e43a69a92b81d9ca058a9"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "8376e72ee597ef3e208e93c0890504d0"
server_key_iv: "bd7298afc553cfec85affe7a16d738ad"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.693529 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "194053f0783e43a69a92b81d9ca058a9"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "8376e72ee597ef3e208e93c0890504d0"
server_key_iv: "bd7298afc553cfec85affe7a16d738ad"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.694933 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:54.695780 9727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.695981 9713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:54.696074 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "194053f0783e43a69a92b81d9ca058a9"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "8376e72ee597ef3e208e93c0890504d0"
server_key_iv: "bd7298afc553cfec85affe7a16d738ad"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.696142 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:54.716038 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:54.716318 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:54.716456 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:54.716678 9713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:54.716992 9713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:54.717024 9713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.717083 9713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:54.717108 9713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.722684 9713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42837
I20260621 05:16:54.722720 9840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42837 every 8 connection(s)
I20260621 05:16:54.723016 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:16:54.726698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 9713
I20260621 05:16:54.726821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:16:54.727027 392 external_mini_cluster.cc:1428] Setting key a95ccd04cfbdc5140aa4b9eaa32f2efa
I20260621 05:16:54.728013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--builtin_ntp_servers=127.0.98.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 05:16:54.728102 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.723694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40197 (local address 127.0.98.62:44825)
0621 05:16:54.723734 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:54.723736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:54.724010 (+ 274us) server_negotiation.cc:407] Connection header received
0621 05:16:54.724517 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:54.724518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:54.724532 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:54.724571 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:54.725193 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.725673 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.726571 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.726785 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.726864 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.727789 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:54.727790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:54.727791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:16:54.727796 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:54.727813 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:16:54.727817 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:54.727877 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:54.727982 (+ 105us) server_negotiation.cc:299] Negotiation successful
0621 05:16:54.728013 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:16:54.728305 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.723704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44825 (local address 127.0.98.1:40197)
0621 05:16:54.723880 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:54.723922 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:54.724407 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:54.724660 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:54.724664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:54.724679 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:54.725100 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.725104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.725772 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.725773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.726471 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.726472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.726887 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.726888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.727013 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.727638 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:16:54.727639 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:54.727676 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:16:54.727703 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:54.727885 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:54.727889 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:54.727891 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:16:54.727954 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:16:54.728056 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260621 05:16:54.729414 9841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44825
I20260621 05:16:54.729514 9841 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:54.729683 9841 heartbeater.cc:511] Master 127.0.98.62:44825 requested a full tablet report, sending...
I20260621 05:16:54.730204 9653 ts_manager.cc:194] Registered new tserver with Master: 194053f0783e43a69a92b81d9ca058a9 (127.0.98.1:42837)
I20260621 05:16:54.730767 9653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40197
W20260621 05:16:54.804677 9844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:54.804888 9844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:54.804921 9844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:54.806006 9844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:54.806067 9844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:54.806109 9844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:54.807367 9844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:54.807614 9844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:54.807919 9844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:54.812450 9852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:54.812431 9850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:54.812500 9849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:54.812768 9844 server_base.cc:1034] running on GCE node
I20260621 05:16:54.812986 9844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:54.813235 9844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:54.814385 9844 hybrid_clock.cc:648] HybridClock initialized: now 1782019014814367 us; error 34 us; skew 500 ppm
I20260621 05:16:54.815659 9844 webserver.cc:460] Webserver started at http://127.0.98.2:44221/ using document root <none> and password file <none>
I20260621 05:16:54.815888 9844 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:54.815958 9844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:54.816087 9844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:54.817126 9844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "685fa7f39b0745518ea844795e81e858"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "a6fe2bc857319f02afbd9476de8088c3"
server_key_iv: "c69501e745c800ee72e2af89860af681"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.817471 9844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "685fa7f39b0745518ea844795e81e858"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "a6fe2bc857319f02afbd9476de8088c3"
server_key_iv: "c69501e745c800ee72e2af89860af681"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.818663 9844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:54.819289 9858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.819447 9844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:54.819538 9844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "685fa7f39b0745518ea844795e81e858"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "a6fe2bc857319f02afbd9476de8088c3"
server_key_iv: "c69501e745c800ee72e2af89860af681"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.819646 9844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:54.832361 9844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:54.832624 9844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:54.832772 9844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:54.832988 9844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:54.833298 9844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:54.833351 9844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.833381 9844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:54.833408 9844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.838778 9844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33345
I20260621 05:16:54.838814 9971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33345 every 8 connection(s)
I20260621 05:16:54.839066 9844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:16:54.842612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 9844
I20260621 05:16:54.842687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:16:54.842890 392 external_mini_cluster.cc:1428] Setting key 8cd401e27d1bb5288597be5cf4aaa2e9
I20260621 05:16:54.843899 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.839666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47607 (local address 127.0.98.62:44825)
0621 05:16:54.839722 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:54.839724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:54.839936 (+ 212us) server_negotiation.cc:407] Connection header received
0621 05:16:54.840430 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:54.840432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:54.840444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:54.840491 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:54.841083 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.841578 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.842511 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.842760 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.842813 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.843669 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:54.843671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:54.843671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:16:54.843675 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:54.843689 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:16:54.843693 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:54.843753 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:54.843828 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:16:54.843849 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:16:54.843935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--builtin_ntp_servers=127.0.98.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 05:16:54.844143 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.839659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44825 (local address 127.0.98.2:47607)
0621 05:16:54.839834 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:54.839861 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:54.840324 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:54.840560 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:54.840563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:54.840570 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:54.840989 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.840993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.841660 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.841661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.842405 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.842407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.842843 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.842844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.843010 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.843514 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:16:54.843514 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:54.843545 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:16:54.843565 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:54.843755 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:54.843759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:54.843763 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:16:54.843831 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:16:54.843937 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260621 05:16:54.844600 9972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44825
I20260621 05:16:54.844683 9972 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:54.844846 9972 heartbeater.cc:511] Master 127.0.98.62:44825 requested a full tablet report, sending...
I20260621 05:16:54.845203 9653 ts_manager.cc:194] Registered new tserver with Master: 685fa7f39b0745518ea844795e81e858 (127.0.98.2:33345)
I20260621 05:16:54.845570 9653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47607
W20260621 05:16:54.919512 9975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:54.919734 9975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:54.919767 9975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:54.920789 9975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:54.920848 9975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:54.920902 9975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:54.922240 9975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:54.922474 9975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:54.922752 9975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:54.927479 9983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:54.927456 9981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:54.927474 9980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:54.927778 9975 server_base.cc:1034] running on GCE node
I20260621 05:16:54.928000 9975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:54.928242 9975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:54.929415 9975 hybrid_clock.cc:648] HybridClock initialized: now 1782019014929404 us; error 27 us; skew 500 ppm
I20260621 05:16:54.930555 9975 webserver.cc:460] Webserver started at http://127.0.98.3:41265/ using document root <none> and password file <none>
I20260621 05:16:54.930800 9975 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:54.930866 9975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:54.930992 9975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:54.932042 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "a47b84f0fb3e40e7bd5d1eba07a35b6c"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "0823cb3320cde54dccb1436389b5ceac"
server_key_iv: "3ffe3c9ca0c54f92ef5ef3064ae11b62"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.932387 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a47b84f0fb3e40e7bd5d1eba07a35b6c"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "0823cb3320cde54dccb1436389b5ceac"
server_key_iv: "3ffe3c9ca0c54f92ef5ef3064ae11b62"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.933619 9975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:16:54.934278 9989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.934444 9975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:54.934541 9975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "a47b84f0fb3e40e7bd5d1eba07a35b6c"
format_stamp: "Formatted at 2026-06-21 05:16:54 on dist-test-slave-czm5"
server_key: "0823cb3320cde54dccb1436389b5ceac"
server_key_iv: "3ffe3c9ca0c54f92ef5ef3064ae11b62"
server_key_version: "encryptionkey@0"
I20260621 05:16:54.934616 9975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:54.963038 9975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:54.963351 9975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:54.963495 9975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:54.963742 9975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:54.964039 9975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:54.964092 9975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.964125 9975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:54.964171 9975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:54.969660 9975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40071
I20260621 05:16:54.969703 10102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40071 every 8 connection(s)
I20260621 05:16:54.969976 9975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:16:54.974432 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.970675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59719 (local address 127.0.98.62:44825)
0621 05:16:54.970734 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:54.970735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:54.970949 (+ 214us) server_negotiation.cc:407] Connection header received
0621 05:16:54.971362 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:54.971363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:54.971370 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:54.971422 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:54.971993 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.972502 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.973397 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.973573 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.973632 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.974235 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:54.974236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:54.974236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:16:54.974239 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:54.974251 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:16:54.974255 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:54.974290 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:54.974359 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 05:16:54.974380 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:16:54.974622 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:54.970667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44825 (local address 127.0.98.3:59719)
0621 05:16:54.970834 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:54.970869 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:54.971288 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:54.971468 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:54.971471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:54.971478 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:54.971922 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.971926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.972580 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.972581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.973318 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:54.973319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:54.973646 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:54.973647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:54.973737 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:54.974124 (+ 387us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:16:54.974124 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:54.974147 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:16:54.974165 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:54.974296 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:54.974298 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:54.974300 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:16:54.974346 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:16:54.974410 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260621 05:16:54.975049 10103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44825
I20260621 05:16:54.975145 10103 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:54.975322 10103 heartbeater.cc:511] Master 127.0.98.62:44825 requested a full tablet report, sending...
I20260621 05:16:54.975715 9653 ts_manager.cc:194] Registered new tserver with Master: a47b84f0fb3e40e7bd5d1eba07a35b6c (127.0.98.3:40071)
I20260621 05:16:54.976064 9653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59719
I20260621 05:16:54.978811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 9975
I20260621 05:16:54.978880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:16:54.979084 392 external_mini_cluster.cc:1428] Setting key 2209e1190ae7cf67e69b6949a39fe486
I20260621 05:16:54.979889 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:16:54.981563 9706 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:54.981133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36998 (local address 127.0.98.62:44825)
0621 05:16:54.981181 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:54.981183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:54.981314 (+ 131us) server_negotiation.cc:407] Connection header received
0621 05:16:54.981359 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:54.981359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:54.981368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:54.981416 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:54.981524 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36998: BlockingRecv error: recv got EOF from 127.0.0.1:36998 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:16:54.981801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 9713
I20260621 05:16:54.986902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 9844
I20260621 05:16:54.991290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 9975
I20260621 05:16:54.995481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 9622
2026-06-21T05:16:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (568 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T05:16:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:16:55Z Disabled control of system clock
I20260621 05:16:55.020643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43669
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:16:55.096040 10114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:55.096227 10114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:55.096246 10114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:55.097348 10114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:55.097388 10114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:55.097400 10114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:55.097412 10114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:55.097460 10114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:55.098702 10114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:55.098924 10114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:55.099182 10114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:55.103591 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:55.103612 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:55.103662 10114 server_base.cc:1034] running on GCE node
W20260621 05:16:55.103634 10122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:55.104046 10114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:55.104303 10114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:55.105434 10114 hybrid_clock.cc:648] HybridClock initialized: now 1782019015105417 us; error 38 us; skew 500 ppm
I20260621 05:16:55.106477 10114 webserver.cc:460] Webserver started at http://127.0.98.62:41995/ using document root <none> and password file <none>
I20260621 05:16:55.106657 10114 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:55.106695 10114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:55.106776 10114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:55.107748 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "0b5da737d83f4a6b9e69feb62597a143"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "c1c549ecb89c68ccd5a649a7432bde3b"
server_key_iv: "9a4fddd5745e281b589aeb33e4c3f705"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.108016 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b5da737d83f4a6b9e69feb62597a143"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "c1c549ecb89c68ccd5a649a7432bde3b"
server_key_iv: "9a4fddd5745e281b589aeb33e4c3f705"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.109083 10114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:16:55.109644 10128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.109814 10114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:55.109901 10114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "0b5da737d83f4a6b9e69feb62597a143"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "c1c549ecb89c68ccd5a649a7432bde3b"
server_key_iv: "9a4fddd5745e281b589aeb33e4c3f705"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.109979 10114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:55.113164 10114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:55.113404 10114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:55.113528 10114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:55.117182 10114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43669
I20260621 05:16:55.117209 10180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43669 every 8 connection(s)
I20260621 05:16:55.117502 10114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:16:55.118084 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:55.120476 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143: Bootstrap starting.
I20260621 05:16:55.120926 10181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:55.121145 10181 log.cc:824] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:55.121671 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143: No bootstrap required, opened a new log
I20260621 05:16:55.123100 10181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5da737d83f4a6b9e69feb62597a143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260621 05:16:55.123231 10181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:55.123266 10181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5da737d83f4a6b9e69feb62597a143, State: Initialized, Role: FOLLOWER
I20260621 05:16:55.123396 10181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5da737d83f4a6b9e69feb62597a143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260621 05:16:55.123469 10181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:16:55.123489 10181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:16:55.123518 10181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:55.124004 10181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5da737d83f4a6b9e69feb62597a143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260621 05:16:55.124081 10181 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b5da737d83f4a6b9e69feb62597a143; no voters:
I20260621 05:16:55.124194 10181 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:16:55.124256 10184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:55.124346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 10114
I20260621 05:16:55.124418 10181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:55.124431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:16:55.124469 10184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [term 1 LEADER]: Becoming Leader. State: Replica: 0b5da737d83f4a6b9e69feb62597a143, State: Running, Role: LEADER
I20260621 05:16:55.124568 10184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5da737d83f4a6b9e69feb62597a143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260621 05:16:55.124627 392 external_mini_cluster.cc:1428] Setting key ebef63c692b642e6ff8c638d6901f411
I20260621 05:16:55.125043 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b5da737d83f4a6b9e69feb62597a143. Latest consensus state: current_term: 1 leader_uuid: "0b5da737d83f4a6b9e69feb62597a143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5da737d83f4a6b9e69feb62597a143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } } }
I20260621 05:16:55.125141 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:55.125206 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b5da737d83f4a6b9e69feb62597a143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5da737d83f4a6b9e69feb62597a143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } } }
I20260621 05:16:55.125284 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143 [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:55.125483 10194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:16:55.125979 10194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:16:55.127357 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:55.125221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.0.98.62:43669)
0621 05:16:55.125607 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:55.125614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:55.125642 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:16:55.125740 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:55.125747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:55.125791 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:55.125909 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:55.126876 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:55.126879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:55.126879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:16:55.126883 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:55.126924 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:16:55.126940 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:55.127044 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:55.127150 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 05:16:55.127203 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":62,"threads_started":1}
I20260621 05:16:55.127715 10194 catalog_manager.cc:1347] Generated new cluster ID: b42c29c246b0474db6373554cfe46b93
I20260621 05:16:55.127779 10194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:16:55.138255 10194 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:16:55.138427 10194 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:16:55.142016 10194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b5da737d83f4a6b9e69feb62597a143: Generated new TSK 0
I20260621 05:16:55.142235 10194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:16:55.145175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--builtin_ntp_servers=127.0.98.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:16:55.221279 10205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:55.221464 10205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:55.221484 10205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:55.222610 10205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:55.222647 10205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:55.222672 10205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:16:55.223987 10205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:55.224232 10205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:55.224489 10205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:55.229418 10210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:55.229439 10211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:55.229456 10213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:55.229949 10205 server_base.cc:1034] running on GCE node
I20260621 05:16:55.230136 10205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:55.230357 10205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:55.231523 10205 hybrid_clock.cc:648] HybridClock initialized: now 1782019015231503 us; error 34 us; skew 500 ppm
I20260621 05:16:55.232771 10205 webserver.cc:460] Webserver started at http://127.0.98.1:38851/ using document root <none> and password file <none>
I20260621 05:16:55.233002 10205 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:55.233073 10205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:55.233220 10205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:55.234170 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "35947beddd034e2a96619c45e793adb1"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "1a5cecc9078af690b38ea73b46fc8b5d"
server_key_iv: "74f911fca3b4c30c7cbd5acf905ffcc9"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.234519 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35947beddd034e2a96619c45e793adb1"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "1a5cecc9078af690b38ea73b46fc8b5d"
server_key_iv: "74f911fca3b4c30c7cbd5acf905ffcc9"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.235934 10205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:16:55.236661 10219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.236838 10205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:55.236932 10205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "35947beddd034e2a96619c45e793adb1"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "1a5cecc9078af690b38ea73b46fc8b5d"
server_key_iv: "74f911fca3b4c30c7cbd5acf905ffcc9"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.237011 10205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:55.240451 10205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:55.240712 10205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:55.240842 10205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:55.241055 10205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:55.241322 10205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:55.241374 10205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.241411 10205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:55.241437 10205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.247067 10205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43937
I20260621 05:16:55.247115 10332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43937 every 8 connection(s)
I20260621 05:16:55.247373 10205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:16:55.249459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 10205
I20260621 05:16:55.249529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:16:55.249728 392 external_mini_cluster.cc:1428] Setting key 3076c6e32da0dcba99a48d116cd6a177
I20260621 05:16:55.250438 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:55.248092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51479 (local address 127.0.98.62:43669)
0621 05:16:55.248147 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:55.248149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:55.248344 (+ 195us) server_negotiation.cc:407] Connection header received
0621 05:16:55.248791 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:55.248792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:55.248798 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:55.248846 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:55.250211 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:55.250212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:55.250212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:16:55.250217 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:55.250232 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:16:55.250236 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:55.250282 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:55.250360 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:16:55.250374 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:16:55.250609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--builtin_ntp_servers=127.0.98.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 05:16:55.250633 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:55.248086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43669 (local address 127.0.98.1:51479)
0621 05:16:55.248242 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:55.248270 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:55.248696 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:55.248944 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:55.248947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:55.248953 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:55.250051 (+ 1098us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:16:55.250052 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:55.250086 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:16:55.250130 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:55.250289 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:55.250293 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:55.250296 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:16:55.250369 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:16:55.250423 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260621 05:16:55.251130 10333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43669
I20260621 05:16:55.251231 10333 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:55.251369 10333 heartbeater.cc:511] Master 127.0.98.62:43669 requested a full tablet report, sending...
I20260621 05:16:55.251873 10145 ts_manager.cc:194] Registered new tserver with Master: 35947beddd034e2a96619c45e793adb1 (127.0.98.1:43937)
W20260621 05:16:55.328442 10336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:55.328604 10336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:55.328624 10336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:55.329599 10336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:55.329628 10336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:55.329650 10336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:55.330794 10336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:55.330945 10336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:55.331141 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:55.335583 10344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:55.335603 10341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:55.335783 10336 server_base.cc:1034] running on GCE node
W20260621 05:16:55.335685 10342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:55.336040 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:55.336278 10336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:55.337445 10336 hybrid_clock.cc:648] HybridClock initialized: now 1782019015337427 us; error 35 us; skew 500 ppm
I20260621 05:16:55.338589 10336 webserver.cc:460] Webserver started at http://127.0.98.2:41845/ using document root <none> and password file <none>
I20260621 05:16:55.338842 10336 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:55.338907 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:55.339035 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:55.340121 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "1181886d097348b795994a88a8b4e6a9"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "5affe37ecd08a7def9c88e681c1102f6"
server_key_iv: "dea823f50b832e33c0b9452cb3a1747a"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.340502 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1181886d097348b795994a88a8b4e6a9"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "5affe37ecd08a7def9c88e681c1102f6"
server_key_iv: "dea823f50b832e33c0b9452cb3a1747a"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.341775 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:55.342389 10350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.342555 10336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:55.342618 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "1181886d097348b795994a88a8b4e6a9"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "5affe37ecd08a7def9c88e681c1102f6"
server_key_iv: "dea823f50b832e33c0b9452cb3a1747a"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.342670 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:55.345613 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:55.345809 10336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:55.345890 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:55.346082 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:55.346278 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:55.346304 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.346323 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:55.346333 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.351747 10336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44423
I20260621 05:16:55.351806 10463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44423 every 8 connection(s)
I20260621 05:16:55.352136 10336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:16:55.354727 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:55.352777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50123 (local address 127.0.98.62:43669)
0621 05:16:55.352839 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:55.352841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:55.353048 (+ 207us) server_negotiation.cc:407] Connection header received
0621 05:16:55.353454 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:55.353455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:55.353459 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:55.353512 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:55.354542 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:55.354544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:55.354544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:16:55.354548 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:55.354562 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:16:55.354567 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:55.354612 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:55.354669 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 05:16:55.354680 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:16:55.354939 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:55.352758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43669 (local address 127.0.98.2:50123)
0621 05:16:55.352936 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:55.352974 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:55.353379 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:55.353557 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:55.353560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:55.353568 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:55.354416 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:16:55.354416 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:55.354440 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:16:55.354465 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:55.354609 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:55.354611 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:55.354613 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:16:55.354676 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:16:55.354716 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260621 05:16:55.355083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 10336
I20260621 05:16:55.355182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:16:55.355362 392 external_mini_cluster.cc:1428] Setting key 70d5c954e7228df4d3e2a442363b28dc
I20260621 05:16:55.355414 10464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43669
I20260621 05:16:55.355491 10464 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:55.355638 10464 heartbeater.cc:511] Master 127.0.98.62:43669 requested a full tablet report, sending...
I20260621 05:16:55.355988 10145 ts_manager.cc:194] Registered new tserver with Master: 1181886d097348b795994a88a8b4e6a9 (127.0.98.2:44423)
I20260621 05:16:55.356356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--builtin_ntp_servers=127.0.98.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:16:55.432340 10467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:55.432521 10467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:55.432542 10467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:55.433692 10467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:55.433743 10467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:55.433764 10467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:55.434978 10467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:55.435245 10467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:55.435515 10467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:55.440138 10473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:55.440140 10475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:55.440169 10472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:55.440253 10467 server_base.cc:1034] running on GCE node
I20260621 05:16:55.440542 10467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:55.440834 10467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:55.441977 10467 hybrid_clock.cc:648] HybridClock initialized: now 1782019015441957 us; error 38 us; skew 500 ppm
I20260621 05:16:55.443145 10467 webserver.cc:460] Webserver started at http://127.0.98.3:33479/ using document root <none> and password file <none>
I20260621 05:16:55.443374 10467 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:55.443414 10467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:55.443538 10467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:55.444552 10467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "a045cd9ab5d44f3ebf0f896f075d187e"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "a4e4844a3dd867740496a1dbca7c6679"
server_key_iv: "6e566192799b6779e73c8708cba90b42"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.444933 10467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a045cd9ab5d44f3ebf0f896f075d187e"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "a4e4844a3dd867740496a1dbca7c6679"
server_key_iv: "6e566192799b6779e73c8708cba90b42"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.446241 10467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:55.446866 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.447039 10467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:55.447104 10467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "a045cd9ab5d44f3ebf0f896f075d187e"
format_stamp: "Formatted at 2026-06-21 05:16:55 on dist-test-slave-czm5"
server_key: "a4e4844a3dd867740496a1dbca7c6679"
server_key_iv: "6e566192799b6779e73c8708cba90b42"
server_key_version: "encryptionkey@0"
I20260621 05:16:55.447153 10467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:55.450208 10467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:55.450429 10467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:55.450515 10467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:55.450659 10467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:55.450927 10467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:55.450979 10467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.451016 10467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:55.451045 10467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:55.456449 10467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38989
I20260621 05:16:55.456524 10594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38989 every 8 connection(s)
I20260621 05:16:55.456881 10467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:16:55.459425 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:55.457526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40689 (local address 127.0.98.62:43669)
0621 05:16:55.457600 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:55.457601 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:55.457786 (+ 185us) server_negotiation.cc:407] Connection header received
0621 05:16:55.458165 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:55.458166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:55.458170 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:55.458223 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:55.459278 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:55.459279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:55.459279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:16:55.459281 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:55.459290 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:16:55.459293 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:55.459328 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:55.459367 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 05:16:55.459378 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:16:55.459635 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:55.457501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43669 (local address 127.0.98.3:40689)
0621 05:16:55.457676 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:55.457709 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:55.458093 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:55.458266 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:55.458269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:55.458276 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:55.459144 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:16:55.459145 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:55.459169 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:16:55.459193 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:55.459328 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:55.459330 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:55.459331 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:16:55.459374 (+ 43us) client_negotiation.cc:240] Negotiation successful
0621 05:16:55.459412 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260621 05:16:55.460088 10595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43669
I20260621 05:16:55.460179 10595 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:55.460323 10595 heartbeater.cc:511] Master 127.0.98.62:43669 requested a full tablet report, sending...
I20260621 05:16:55.460420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 10467
I20260621 05:16:55.460511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:16:55.460712 392 external_mini_cluster.cc:1428] Setting key 8eceae6017f24d5e2ebc8bf1e0564c53
I20260621 05:16:55.460718 10145 ts_manager.cc:194] Registered new tserver with Master: a045cd9ab5d44f3ebf0f896f075d187e (127.0.98.3:38989)
I20260621 05:16:55.461443 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:16:55.463052 10195 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:16:55.462662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.0.98.62:43669)
0621 05:16:55.462734 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:55.462736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:55.462790 (+ 54us) server_negotiation.cc:407] Connection header received
0621 05:16:55.462845 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:55.462845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:55.462848 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:55.462891 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:55.462996 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60882: BlockingRecv error: recv got EOF from 127.0.0.1:60882 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:16:55.463106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 10205
I20260621 05:16:55.468040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 10336
I20260621 05:16:55.472012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 10467
I20260621 05:16:55.476234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 10114
2026-06-21T05:16:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (481 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:16:55 dist-test-slave-czm5 krb5kdc[10605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:16:55 dist-test-slave-czm5 krb5kdc[10605](info): set up 2 sockets
Jun 21 05:16:55 dist-test-slave-czm5 krb5kdc[10605](info): commencing operation
krb5kdc: starting...
W20260621 05:16:56.483916 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:16:56 dist-test-slave-czm5 krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:16:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:16:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:16:56.588285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:56.662575 10618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:56.662824 10618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:56.662864 10618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:56.664403 10618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:16:56.664466 10618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:56.664490 10618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:56.664520 10618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:16:56.664546 10618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:16:56.665943 10618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:56.666189 10618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:56.666383 10618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:56.670925 10626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:56.670943 10624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:56.670960 10618 server_base.cc:1034] running on GCE node
W20260621 05:16:56.670943 10623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:56.671352 10618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:56.671672 10618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:56.672823 10618 hybrid_clock.cc:648] HybridClock initialized: now 1782019016672810 us; error 27 us; skew 500 ppm
Jun 21 05:16:56 dist-test-slave-czm5 krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:56.675065 10618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:16:56.675495 10618 webserver.cc:460] Webserver started at http://127.0.98.62:42443/ using document root <none> and password file <none>
I20260621 05:16:56.675770 10618 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:56.675836 10618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:56.675982 10618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:56.676920 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "551f74bc74134aacab2e3df929c96cda"
format_stamp: "Formatted at 2026-06-21 05:16:56 on dist-test-slave-czm5"
server_key: "97ef930afe9605e4d9446758a02fad48"
server_key_iv: "d0c98470d7627bb8bb7333c3f73567af"
server_key_version: "encryptionkey@0"
I20260621 05:16:56.677240 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "551f74bc74134aacab2e3df929c96cda"
format_stamp: "Formatted at 2026-06-21 05:16:56 on dist-test-slave-czm5"
server_key: "97ef930afe9605e4d9446758a02fad48"
server_key_iv: "d0c98470d7627bb8bb7333c3f73567af"
server_key_version: "encryptionkey@0"
I20260621 05:16:56.678464 10618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:16:56.679121 10633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:56.679309 10618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:56.679394 10618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "551f74bc74134aacab2e3df929c96cda"
format_stamp: "Formatted at 2026-06-21 05:16:56 on dist-test-slave-czm5"
server_key: "97ef930afe9605e4d9446758a02fad48"
server_key_iv: "d0c98470d7627bb8bb7333c3f73567af"
server_key_version: "encryptionkey@0"
I20260621 05:16:56.679472 10618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:56.701145 10618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:56.702702 10618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:56.702852 10618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:56.706461 10618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43361
I20260621 05:16:56.706491 10685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43361 every 8 connection(s)
I20260621 05:16:56.706785 10618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:16:56.707624 10686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:56.709636 10686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda: Bootstrap starting.
I20260621 05:16:56.710198 10686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:56.710511 10686 log.cc:824] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:56.711161 10686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda: No bootstrap required, opened a new log
I20260621 05:16:56.712409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 10618
I20260621 05:16:56.712509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:16:56.712702 392 external_mini_cluster.cc:1428] Setting key bdc5b920d4bc2fcef36e4d728a058762
I20260621 05:16:56.713186 10686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551f74bc74134aacab2e3df929c96cda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } }
I20260621 05:16:56.713372 10686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:56.713451 10686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551f74bc74134aacab2e3df929c96cda, State: Initialized, Role: FOLLOWER
I20260621 05:16:56.713563 10686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551f74bc74134aacab2e3df929c96cda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } }
I20260621 05:16:56.713629 10686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:16:56.713672 10686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:16:56.713754 10686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:56.714447 10686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551f74bc74134aacab2e3df929c96cda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } }
I20260621 05:16:56.714548 10686 leader_election.cc:302] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 551f74bc74134aacab2e3df929c96cda; no voters:
I20260621 05:16:56.714686 10686 leader_election.cc:288] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:16:56.714725 10691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:56.714810 10691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [term 1 LEADER]: Becoming Leader. State: Replica: 551f74bc74134aacab2e3df929c96cda, State: Running, Role: LEADER
I20260621 05:16:56.714900 10686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:16:56.714875 10691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551f74bc74134aacab2e3df929c96cda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } }
I20260621 05:16:56.715221 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "551f74bc74134aacab2e3df929c96cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551f74bc74134aacab2e3df929c96cda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } }
I20260621 05:16:56.715371 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:56.715555 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 551f74bc74134aacab2e3df929c96cda. Latest consensus state: current_term: 1 leader_uuid: "551f74bc74134aacab2e3df929c96cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551f74bc74134aacab2e3df929c96cda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } }
I20260621 05:16:56.715710 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda [sys.catalog]: This master's current role is: LEADER
I20260621 05:16:56.715981 10699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:16:56.716492 10699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:16:56.717892 10699 catalog_manager.cc:1347] Generated new cluster ID: a2ac4a906969437e8011117e1af48a13
I20260621 05:16:56.717936 10699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:16:56 dist-test-slave-czm5 krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:56.722096 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:56.713170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.0.98.62:43361)
0621 05:16:56.713353 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:56.713360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:56.713417 (+ 57us) server_negotiation.cc:407] Connection header received
0621 05:16:56.713522 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:56.713528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:56.713577 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:56.713735 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:56.714262 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:56.715278 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:56.716395 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:56.716907 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:56.717004 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:56.719158 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:56.719161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:56.719162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:56.719172 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:56.720702 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:56.721082 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:56.721084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:56.721084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:56.721187 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:56.721384 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:56.721385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:56.721385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:56.721586 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:56.721747 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:56.721811 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 05:16:56.721944 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260621 05:16:56.749928 10699 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:16:56.750581 10699 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:16:56.764593 10699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda: Generated new TSK 0
I20260621 05:16:56.764746 10699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:16:56.793701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43361
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:56.869632 10712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:56.869797 10712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:56.869817 10712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:56.870850 10712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:56.870906 10712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:56.870954 10712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:16:56.872400 10712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:56.872627 10712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:56.872938 10712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:56.877604 10720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:56.877635 10717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:56.877610 10718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:56.877903 10712 server_base.cc:1034] running on GCE node
I20260621 05:16:56.878136 10712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:56.878360 10712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:56.879529 10712 hybrid_clock.cc:648] HybridClock initialized: now 1782019016879517 us; error 41 us; skew 500 ppm
Jun 21 05:16:56 dist-test-slave-czm5 krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:56.881840 10712 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:16:56.882326 10712 webserver.cc:460] Webserver started at http://127.0.98.1:34793/ using document root <none> and password file <none>
I20260621 05:16:56.882493 10712 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:56.882530 10712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:56.882613 10712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:56.883550 10712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebf5c11c4a5148fca343e44816f70c1f"
format_stamp: "Formatted at 2026-06-21 05:16:56 on dist-test-slave-czm5"
server_key: "2bf30c88ca2af280b4142147b27d9bff"
server_key_iv: "9d7978d62c01967bc4c37941e9c497d8"
server_key_version: "encryptionkey@0"
I20260621 05:16:56.883889 10712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebf5c11c4a5148fca343e44816f70c1f"
format_stamp: "Formatted at 2026-06-21 05:16:56 on dist-test-slave-czm5"
server_key: "2bf30c88ca2af280b4142147b27d9bff"
server_key_iv: "9d7978d62c01967bc4c37941e9c497d8"
server_key_version: "encryptionkey@0"
I20260621 05:16:56.885036 10712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:16:56.885651 10727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:56.885795 10712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:56.885877 10712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "ebf5c11c4a5148fca343e44816f70c1f"
format_stamp: "Formatted at 2026-06-21 05:16:56 on dist-test-slave-czm5"
server_key: "2bf30c88ca2af280b4142147b27d9bff"
server_key_iv: "9d7978d62c01967bc4c37941e9c497d8"
server_key_version: "encryptionkey@0"
I20260621 05:16:56.885926 10712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:56.901481 10712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:56.903076 10712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:56.903175 10712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:56.903327 10712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:56.903543 10712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:56.903571 10712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:56.903589 10712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:56.903602 10712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:56.908927 10712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43191
I20260621 05:16:56.908998 10840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43191 every 8 connection(s)
I20260621 05:16:56.909256 10712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:16:56 dist-test-slave-czm5 krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:56.917240 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:56.909873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35151 (local address 127.0.98.62:43361)
0621 05:16:56.909910 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:56.909912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:56.910082 (+ 170us) server_negotiation.cc:407] Connection header received
0621 05:16:56.910475 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:56.910476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:56.910487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:56.910536 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:56.911314 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:56.911867 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:56.912775 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:56.912957 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:56.913005 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:56.914864 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:56.914865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:56.914865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:56.914871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:56.916178 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:56.916507 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:56.916508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:56.916509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:56.916565 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:56.916762 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:56.916762 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:56.916763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:56.916863 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:56.916910 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:56.917132 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 05:16:56.917184 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:16:56.917356 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:56.909845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43361 (local address 127.0.98.1:35151)
0621 05:16:56.909980 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:56.910016 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:56.910401 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:56.910579 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:56.910582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:56.911068 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:56.911245 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:56.911248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:56.911937 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:56.911938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:56.912700 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:56.912702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:56.913024 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:56.913024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:56.913110 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:56.913503 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:56.913509 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:56.914787 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:56.916256 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:56.916261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:56.916263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:56.916442 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:56.916621 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:56.916621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:56.916622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:56.916712 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:56.916923 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:56.916927 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:56.917021 (+ 94us) client_negotiation.cc:769] Sending connection context
0621 05:16:56.917094 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:16:56.917202 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260621 05:16:56.917826 10841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43361
I20260621 05:16:56.917904 10841 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:56.918068 10841 heartbeater.cc:511] Master 127.0.98.62:43361 requested a full tablet report, sending...
I20260621 05:16:56.918097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 10712
I20260621 05:16:56.918191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:16:56.918392 392 external_mini_cluster.cc:1428] Setting key 01d926a2e000d8aa9e3e0b6d9857b1d5
I20260621 05:16:56.918574 10650 ts_manager.cc:194] Registered new tserver with Master: ebf5c11c4a5148fca343e44816f70c1f (127.0.98.1:43191)
I20260621 05:16:56.919394 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35151
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:16:56.940248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43361
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:57.014912 10846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:57.015110 10846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:57.015132 10846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:57.016371 10846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:57.016412 10846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:57.016462 10846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:16:57.017913 10846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:57.018129 10846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:57.018337 10846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:16:57.022907 10846 server_base.cc:1034] running on GCE node
W20260621 05:16:57.022835 10851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:57.022845 10854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:57.022861 10852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:57.023257 10846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:57.023532 10846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:57.024688 10846 hybrid_clock.cc:648] HybridClock initialized: now 1782019017024671 us; error 29 us; skew 500 ppm
Jun 21 05:16:57 dist-test-slave-czm5 krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:57.026872 10846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:16:57.027218 10846 webserver.cc:460] Webserver started at http://127.0.98.2:33931/ using document root <none> and password file <none>
I20260621 05:16:57.027376 10846 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:57.027412 10846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:57.027488 10846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:57.028400 10846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "ffd92c211a9d45b58a77e7024f1167de"
format_stamp: "Formatted at 2026-06-21 05:16:57 on dist-test-slave-czm5"
server_key: "03c567c43a9390981d8ed91c2ae7efd2"
server_key_iv: "443406c117af32f048deb36e4e2a033d"
server_key_version: "encryptionkey@0"
I20260621 05:16:57.028667 10846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ffd92c211a9d45b58a77e7024f1167de"
format_stamp: "Formatted at 2026-06-21 05:16:57 on dist-test-slave-czm5"
server_key: "03c567c43a9390981d8ed91c2ae7efd2"
server_key_iv: "443406c117af32f048deb36e4e2a033d"
server_key_version: "encryptionkey@0"
I20260621 05:16:57.029790 10846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:16:57.030419 10861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:57.030562 10846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:16:57.030661 10846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "ffd92c211a9d45b58a77e7024f1167de"
format_stamp: "Formatted at 2026-06-21 05:16:57 on dist-test-slave-czm5"
server_key: "03c567c43a9390981d8ed91c2ae7efd2"
server_key_iv: "443406c117af32f048deb36e4e2a033d"
server_key_version: "encryptionkey@0"
I20260621 05:16:57.030714 10846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:57.041271 10846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:57.042807 10846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:57.042906 10846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:57.043063 10846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:57.043339 10846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:57.043371 10846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:57.043391 10846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:57.043452 10846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:57.048781 10846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44285
I20260621 05:16:57.048849 10974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44285 every 8 connection(s)
I20260621 05:16:57.049098 10846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:16:57.053915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 10846
I20260621 05:16:57.054004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
Jun 21 05:16:57 dist-test-slave-czm5 krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:57.054180 392 external_mini_cluster.cc:1428] Setting key 29ef4dee10b9bab237a4f33600cdc5f8
I20260621 05:16:57.057432 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.049742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57277 (local address 127.0.98.62:43361)
0621 05:16:57.049791 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.049793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.049950 (+ 157us) server_negotiation.cc:407] Connection header received
0621 05:16:57.050302 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.050302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.050312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.050358 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:57.051083 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.051514 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.052427 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.052602 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.052651 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.054653 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:57.054655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:57.054655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:57.054661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:57.056138 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:57.056531 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:57.056532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:57.056533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:57.056586 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:57.056838 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:57.056839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:57.056840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:57.056950 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:57.057023 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.057271 (+ 248us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.057345 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:16:57.057595 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.049718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43361 (local address 127.0.98.2:57277)
0621 05:16:57.049844 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:57.049871 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:57.050230 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:57.050400 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:57.050402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:57.050858 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:57.051013 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.051016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.051589 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.051590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.052352 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.052354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.052676 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.052677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.052766 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.053161 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:57.053168 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:57.054566 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:57.056242 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:57.056248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:57.056250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:57.056455 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:57.056662 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:57.056663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:57.056663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:57.056767 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:57.057015 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:57.057020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:57.057149 (+ 129us) client_negotiation.cc:769] Sending connection context
0621 05:16:57.057250 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 05:16:57.057388 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260621 05:16:57.058233 10975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43361
I20260621 05:16:57.058322 10975 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:57.058467 10975 heartbeater.cc:511] Master 127.0.98.62:43361 requested a full tablet report, sending...
I20260621 05:16:57.058854 10650 ts_manager.cc:194] Registered new tserver with Master: ffd92c211a9d45b58a77e7024f1167de (127.0.98.2:44285)
I20260621 05:16:57.059234 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57277
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:16:57.076547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43361
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:16:57.151854 10980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:16:57.152052 10980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:16:57.152073 10980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:16:57.153199 10980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:16:57.153239 10980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:16:57.153291 10980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:16:57.154646 10980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:16:57.154868 10980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:16:57.155126 10980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:16:57.159845 10986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:57.159816 10985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:16:57.159816 10988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:16:57.160019 10980 server_base.cc:1034] running on GCE node
I20260621 05:16:57.160233 10980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:16:57.160467 10980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:16:57.161623 10980 hybrid_clock.cc:648] HybridClock initialized: now 1782019017161601 us; error 41 us; skew 500 ppm
Jun 21 05:16:57 dist-test-slave-czm5 krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:16:57.164198 10980 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:16:57.164585 10980 webserver.cc:460] Webserver started at http://127.0.98.3:46491/ using document root <none> and password file <none>
I20260621 05:16:57.164817 10980 fs_manager.cc:359] Metadata directory not provided
I20260621 05:16:57.164887 10980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:16:57.165033 10980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:16:57.166009 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "03e0c2c807474eedbde39803bacbe51f"
format_stamp: "Formatted at 2026-06-21 05:16:57 on dist-test-slave-czm5"
server_key: "b4c08790b5503f891899a8d7d9828ecf"
server_key_iv: "c1dd5c6feedf2a360b6e094048c1b6c9"
server_key_version: "encryptionkey@0"
I20260621 05:16:57.166356 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "03e0c2c807474eedbde39803bacbe51f"
format_stamp: "Formatted at 2026-06-21 05:16:57 on dist-test-slave-czm5"
server_key: "b4c08790b5503f891899a8d7d9828ecf"
server_key_iv: "c1dd5c6feedf2a360b6e094048c1b6c9"
server_key_version: "encryptionkey@0"
I20260621 05:16:57.167608 10980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:16:57.168391 10995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:57.168545 10980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:16:57.168638 10980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "03e0c2c807474eedbde39803bacbe51f"
format_stamp: "Formatted at 2026-06-21 05:16:57 on dist-test-slave-czm5"
server_key: "b4c08790b5503f891899a8d7d9828ecf"
server_key_iv: "c1dd5c6feedf2a360b6e094048c1b6c9"
server_key_version: "encryptionkey@0"
I20260621 05:16:57.168727 10980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:16:57.188442 10980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:16:57.190085 10980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:16:57.190228 10980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:16:57.190423 10980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:16:57.190685 10980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:16:57.190733 10980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:57.190769 10980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:16:57.190794 10980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:16:57.196467 10980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36105
I20260621 05:16:57.196516 11108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36105 every 8 connection(s)
I20260621 05:16:57.196813 10980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:16:57.201078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 10980
I20260621 05:16:57.201181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:16:57.201363 392 external_mini_cluster.cc:1428] Setting key 9eeaadba9f7a15a332b382fdf3a8a4e5
Jun 21 05:16:57 dist-test-slave-czm5 krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:16:57.205042 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.197461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60977 (local address 127.0.98.62:43361)
0621 05:16:57.197567 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.197569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.197730 (+ 161us) server_negotiation.cc:407] Connection header received
0621 05:16:57.198145 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.198146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.198156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.198204 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:57.199053 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.199482 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.200411 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.200584 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.200633 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.202859 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:57.202860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:57.202860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:57.202866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:57.204005 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:57.204350 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:57.204351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:57.204351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:57.204398 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:57.204592 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:57.204593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:57.204593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:57.204686 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:57.204740 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.204935 (+ 195us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.204985 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 05:16:57.205202 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.197437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43361 (local address 127.0.98.3:60977)
0621 05:16:57.197622 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:57.197655 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:57.198076 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:57.198274 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:57.198277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:57.198807 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:16:57.198971 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.198975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.199550 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.199550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.200336 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.200339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.200649 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.200650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.200741 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.201214 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:16:57.201222 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:16:57.202785 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:16:57.204074 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:57.204079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:57.204080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:57.204288 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:57.204451 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:16:57.204452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:16:57.204452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:16:57.204535 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:16:57.204753 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:16:57.204756 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:16:57.204851 (+ 95us) client_negotiation.cc:769] Sending connection context
0621 05:16:57.204923 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:16:57.205021 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260621 05:16:57.205585 11109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43361
I20260621 05:16:57.205680 11109 heartbeater.cc:464] Registering TS with master...
I20260621 05:16:57.205825 11109 heartbeater.cc:511] Master 127.0.98.62:43361 requested a full tablet report, sending...
I20260621 05:16:57.206194 10650 ts_manager.cc:194] Registered new tserver with Master: 03e0c2c807474eedbde39803bacbe51f (127.0.98.3:36105)
I20260621 05:16:57.206514 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60977
I20260621 05:16:57.213011 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:16:57.219641 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.214267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.0.98.62:43361)
0621 05:16:57.214325 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.214327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.214411 (+ 84us) server_negotiation.cc:407] Connection header received
0621 05:16:57.214478 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.214479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.214487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.214535 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:16:57.215117 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.215556 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.216362 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.216526 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.216599 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.217548 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:16:57.217549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:16:57.217549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:16:57.217555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:16:57.218581 (+ 1026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:57.218908 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:57.218909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:57.218909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:57.218956 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:16:57.219159 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:16:57.219160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:16:57.219160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:16:57.219280 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:16:57.219373 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.219472 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.219537 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:16:57.220778 10650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:16:57.222218 10650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:16:57.227824 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.224198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.0.98.3:36105)
0621 05:16:57.224469 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.224473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.224481 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:57.224515 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.224516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.224526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.224604 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:57.224743 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.225680 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.226713 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.227392 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.227461 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.227506 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.227714 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.227751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":68,"threads_started":1}
I20260621 05:16:57.227825 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.224154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36105 (local address 127.0.0.1:40808)
0621 05:16:57.224328 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:57.224340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:57.224393 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:57.224607 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:57.224609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:57.224611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:57.224668 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.224670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.225790 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.225790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.226631 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.226633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.227499 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.227500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.227602 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.227615 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:16:57.227703 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:16:57.227750 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":35,"thread_start_us":42,"threads_started":1}
I20260621 05:16:57.228032 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.224308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44285 (local address 127.0.0.1:55150)
0621 05:16:57.224638 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:57.224655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:57.224707 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:57.224849 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:57.224850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:57.224851 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:57.224897 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.224898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.225770 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.225771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.226893 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.226895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.227800 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.227801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.227860 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.227863 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:16:57.227916 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:16:57.227958 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":1,"thread_start_us":37,"threads_started":1}
I20260621 05:16:57.228277 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.224146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43191 (local address 127.0.0.1:39308)
0621 05:16:57.224286 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:57.224318 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:57.224369 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:57.224962 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:57.224963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:57.224965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:57.225000 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.225001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.226141 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.226141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.227119 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.227121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.228036 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.228037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.228098 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.228102 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:16:57.228178 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:16:57.228218 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260621 05:16:57.228313 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.224293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.0.98.1:43191)
0621 05:16:57.224715 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.224719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.224727 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:57.224764 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.224766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.224780 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.225075 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:57.225134 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.226038 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.227197 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.227956 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.228034 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.228083 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.228189 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.228224 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":78,"threads_started":1}
I20260621 05:16:57.228405 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.224470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.0.98.2:44285)
0621 05:16:57.224620 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.224625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.224697 (+ 72us) server_negotiation.cc:407] Connection header received
0621 05:16:57.224765 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.224767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.224780 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.224846 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:57.224948 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.225665 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.226984 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.227609 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.227710 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.227768 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.228254 (+ 486us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.228305 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260621 05:16:57.228837 11043 tablet_service.cc:1467] Processing CreateTablet for tablet e0d556b4cd854560b9a4f51391eb01db (DEFAULT_TABLE table=test-table [id=4b526a4538724777b337288e9a33b2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:57.229017 10909 tablet_service.cc:1467] Processing CreateTablet for tablet e0d556b4cd854560b9a4f51391eb01db (DEFAULT_TABLE table=test-table [id=4b526a4538724777b337288e9a33b2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:57.229157 11043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d556b4cd854560b9a4f51391eb01db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:57.229316 10909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d556b4cd854560b9a4f51391eb01db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:57.231310 11123 tablet_bootstrap.cc:492] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Bootstrap starting.
I20260621 05:16:57.231583 11124 tablet_bootstrap.cc:492] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Bootstrap starting.
I20260621 05:16:57.231648 10775 tablet_service.cc:1467] Processing CreateTablet for tablet e0d556b4cd854560b9a4f51391eb01db (DEFAULT_TABLE table=test-table [id=4b526a4538724777b337288e9a33b2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:16:57.231940 10775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d556b4cd854560b9a4f51391eb01db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:16:57.232008 11123 tablet_bootstrap.cc:654] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:57.232164 11124 tablet_bootstrap.cc:654] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:57.232612 11123 log.cc:824] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:57.232754 11124 log.cc:824] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:57.233410 11123 tablet_bootstrap.cc:492] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: No bootstrap required, opened a new log
I20260621 05:16:57.233475 11123 ts_tablet_manager.cc:1397] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:16:57.234702 11127 tablet_bootstrap.cc:492] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f: Bootstrap starting.
I20260621 05:16:57.234907 11123 raft_consensus.cc:348] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.235119 11123 raft_consensus.cc:374] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:57.235170 11123 raft_consensus.cc:729] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e0c2c807474eedbde39803bacbe51f, State: Initialized, Role: FOLLOWER
I20260621 05:16:57.235299 11123 consensus_queue.cc:260] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.235373 11127 tablet_bootstrap.cc:654] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f: Neither blocks nor log segments found. Creating new log.
I20260621 05:16:57.235504 11124 tablet_bootstrap.cc:492] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: No bootstrap required, opened a new log
I20260621 05:16:57.235562 11123 ts_tablet_manager.cc:1428] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:16:57.235571 11124 ts_tablet_manager.cc:1397] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260621 05:16:57.235711 11109 heartbeater.cc:503] Master 127.0.98.62:43361 was elected leader, sending a full tablet report...
I20260621 05:16:57.235944 11127 log.cc:824] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:16:57.236722 11127 tablet_bootstrap.cc:492] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f: No bootstrap required, opened a new log
I20260621 05:16:57.236791 11127 ts_tablet_manager.cc:1397] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:16:57.237131 11124 raft_consensus.cc:348] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.237272 11124 raft_consensus.cc:374] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:57.237311 11124 raft_consensus.cc:729] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd92c211a9d45b58a77e7024f1167de, State: Initialized, Role: FOLLOWER
I20260621 05:16:57.237406 11124 consensus_queue.cc:260] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.237668 10975 heartbeater.cc:503] Master 127.0.98.62:43361 was elected leader, sending a full tablet report...
I20260621 05:16:57.237794 11124 ts_tablet_manager.cc:1428] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:16:57.238418 11127 raft_consensus.cc:348] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.238564 11127 raft_consensus.cc:374] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:16:57.238595 11127 raft_consensus.cc:729] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf5c11c4a5148fca343e44816f70c1f, State: Initialized, Role: FOLLOWER
I20260621 05:16:57.238687 11127 consensus_queue.cc:260] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.238940 11127 ts_tablet_manager.cc:1428] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:16:57.238953 10841 heartbeater.cc:503] Master 127.0.98.62:43361 was elected leader, sending a full tablet report...
W20260621 05:16:57.299674 10976 tablet.cc:2369] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:16:57.410095 10842 tablet.cc:2369] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:16:57.435449 11128 raft_consensus.cc:482] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:16:57.435552 11128 raft_consensus.cc:504] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.435959 11128 leader_election.cc:288] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebf5c11c4a5148fca343e44816f70c1f (127.0.98.1:43191), ffd92c211a9d45b58a77e7024f1167de (127.0.98.2:44285)
I20260621 05:16:57.438659 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.436164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43191 (local address 127.0.98.3:35447)
0621 05:16:57.436217 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:57.436228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:57.436269 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:57.436468 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:57.436469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:57.436470 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:57.436525 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.436526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.437174 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.437175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.437816 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.437818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.438394 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.438395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.438473 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.438476 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:16:57.438550 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:16:57.438590 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:16:57.438659 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.436243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35447 (local address 127.0.98.1:43191)
0621 05:16:57.436312 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.436314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.436323 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:16:57.436351 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.436352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.436362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.436448 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:57.436605 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.437086 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.437909 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.438273 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.438394 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.438405 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.438565 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.438599 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:16:57.438833 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.436297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44285 (local address 127.0.98.3:50161)
0621 05:16:57.436454 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:57.436464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:16:57.436516 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:16:57.436674 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:16:57.436675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:16:57.436677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:16:57.436755 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.436757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.437352 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.437353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.438007 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:16:57.438008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.438576 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.438577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:16:57.438653 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.438656 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:16:57.438722 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:16:57.438759 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260621 05:16:57.438824 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.436371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50161 (local address 127.0.98.2:44285)
0621 05:16:57.436427 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.436430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.436511 (+ 81us) server_negotiation.cc:407] Connection header received
0621 05:16:57.436571 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.436572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.436585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.436669 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:16:57.436842 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.437277 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.438092 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.438496 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.438576 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.438584 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.438728 (+ 144us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.438763 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:16:57.439334 10795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d556b4cd854560b9a4f51391eb01db" candidate_uuid: "03e0c2c807474eedbde39803bacbe51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf5c11c4a5148fca343e44816f70c1f" is_pre_election: true
I20260621 05:16:57.439334 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d556b4cd854560b9a4f51391eb01db" candidate_uuid: "03e0c2c807474eedbde39803bacbe51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd92c211a9d45b58a77e7024f1167de" is_pre_election: true
I20260621 05:16:57.439478 10929 raft_consensus.cc:2454] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e0c2c807474eedbde39803bacbe51f in term 0.
I20260621 05:16:57.439478 10795 raft_consensus.cc:2454] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e0c2c807474eedbde39803bacbe51f in term 0.
I20260621 05:16:57.439746 10997 leader_election.cc:302] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e0c2c807474eedbde39803bacbe51f, ffd92c211a9d45b58a77e7024f1167de; no voters:
I20260621 05:16:57.439904 11128 raft_consensus.cc:2781] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:16:57.439970 11128 raft_consensus.cc:482] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:16:57.439985 11128 raft_consensus.cc:3037] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:57.440616 11128 raft_consensus.cc:504] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.440730 11128 leader_election.cc:288] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [CANDIDATE]: Term 1 election: Requested vote from peers ebf5c11c4a5148fca343e44816f70c1f (127.0.98.1:43191), ffd92c211a9d45b58a77e7024f1167de (127.0.98.2:44285)
I20260621 05:16:57.440904 10795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d556b4cd854560b9a4f51391eb01db" candidate_uuid: "03e0c2c807474eedbde39803bacbe51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf5c11c4a5148fca343e44816f70c1f"
I20260621 05:16:57.440939 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d556b4cd854560b9a4f51391eb01db" candidate_uuid: "03e0c2c807474eedbde39803bacbe51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd92c211a9d45b58a77e7024f1167de"
I20260621 05:16:57.440975 10795 raft_consensus.cc:3037] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:57.441031 10929 raft_consensus.cc:3037] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:16:57.441646 10795 raft_consensus.cc:2454] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e0c2c807474eedbde39803bacbe51f in term 1.
I20260621 05:16:57.441754 10929 raft_consensus.cc:2454] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e0c2c807474eedbde39803bacbe51f in term 1.
I20260621 05:16:57.441804 10997 leader_election.cc:302] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e0c2c807474eedbde39803bacbe51f, ebf5c11c4a5148fca343e44816f70c1f; no voters:
I20260621 05:16:57.441890 11128 raft_consensus.cc:2781] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:16:57.442054 11128 raft_consensus.cc:686] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 1 LEADER]: Becoming Leader. State: Replica: 03e0c2c807474eedbde39803bacbe51f, State: Running, Role: LEADER
I20260621 05:16:57.442162 11128 consensus_queue.cc:237] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } }
I20260621 05:16:57.442966 10649 catalog_manager.cc:5495] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e0c2c807474eedbde39803bacbe51f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "03e0c2c807474eedbde39803bacbe51f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e0c2c807474eedbde39803bacbe51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } health_report { overall_health: HEALTHY } } }
W20260621 05:16:57.447291 11110 tablet.cc:2369] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:16:57.458889 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.456732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.0.98.3:36105)
0621 05:16:57.456771 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:16:57.456773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:16:57.456781 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:16:57.456818 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:16:57.456819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:16:57.456835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:16:57.456893 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:16:57.457025 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.457417 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.458137 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:16:57.458334 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:16:57.458405 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:16:57.458590 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:16:57.458677 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:16:57.458766 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:16:57.458788 (+ 22us) server_negotiation.cc:299] Negotiation successful
0621 05:16:57.458806 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:16:57.460443 10929 raft_consensus.cc:1261] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 1 FOLLOWER]: Refusing update from remote peer 03e0c2c807474eedbde39803bacbe51f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:57.460469 10795 raft_consensus.cc:1261] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 1 FOLLOWER]: Refusing update from remote peer 03e0c2c807474eedbde39803bacbe51f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:16:57.460765 11133 consensus_queue.cc:1035] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd92c211a9d45b58a77e7024f1167de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:57.460860 11128 consensus_queue.cc:1035] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf5c11c4a5148fca343e44816f70c1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:16:57.462909 11136 mvcc.cc:205] Tried to move back new op lower bound from 7299149895515459584 to 7299149895443664896. Current Snapshot: MvccSnapshot[applied={T|T < 7299149895515459584}]
I20260621 05:16:57.465003 10649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:57.465085 10649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:16:57.465668 10649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 551f74bc74134aacab2e3df929c96cda: Sending DeleteTablet for 3 replicas of tablet e0d556b4cd854560b9a4f51391eb01db
I20260621 05:16:57.465931 10775 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d556b4cd854560b9a4f51391eb01db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39308
I20260621 05:16:57.465982 10909 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d556b4cd854560b9a4f51391eb01db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55150
I20260621 05:16:57.466133 11043 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d556b4cd854560b9a4f51391eb01db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:16:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40808
I20260621 05:16:57.466138 11146 tablet_replica.cc:330] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f: stopping tablet replica
I20260621 05:16:57.466157 11147 tablet_replica.cc:330] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: stopping tablet replica
I20260621 05:16:57.466229 11146 raft_consensus.cc:2229] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:57.466229 11147 raft_consensus.cc:2229] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:16:57.466313 11147 raft_consensus.cc:2258] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:57.466312 11146 raft_consensus.cc:2258] T e0d556b4cd854560b9a4f51391eb01db P ebf5c11c4a5148fca343e44816f70c1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:57.466393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 10712
I20260621 05:16:57.467864 11147 ts_tablet_manager.cc:1905] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:16:57.469224 11147 ts_tablet_manager.cc:1918] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:57.469309 11147 log.cc:1196] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/e0d556b4cd854560b9a4f51391eb01db
I20260621 05:16:57.469547 11147 ts_tablet_manager.cc:1939] T e0d556b4cd854560b9a4f51391eb01db P ffd92c211a9d45b58a77e7024f1167de: Deleting consensus metadata
I20260621 05:16:57.469902 10635 catalog_manager.cc:4841] TS ffd92c211a9d45b58a77e7024f1167de (127.0.98.2:44285): tablet e0d556b4cd854560b9a4f51391eb01db (table test-table [id=4b526a4538724777b337288e9a33b2ed]) successfully deleted
I20260621 05:16:57.471729 11148 tablet_replica.cc:330] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: stopping tablet replica
I20260621 05:16:57.471820 11148 raft_consensus.cc:2229] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:16:57.471896 11148 raft_consensus.cc:2258] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:16:57.472214 11148 ts_tablet_manager.cc:1905] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 05:16:57.472718 10635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43191 (error 108)
I20260621 05:16:57.472954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 10846
I20260621 05:16:57.473253 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:16:57.473140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43191 (local address 127.0.0.1:39314)
0621 05:16:57.473187 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:16:57.473220 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:16:57.473435 10635 catalog_manager.cc:4584] TS ebf5c11c4a5148fca343e44816f70c1f (127.0.98.1:43191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0d556b4cd854560b9a4f51391eb01db: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43191: connect: Connection refused (error 111)
I20260621 05:16:57.473757 11148 ts_tablet_manager.cc:1918] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:16:57.473922 11148 log.cc:1196] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/e0d556b4cd854560b9a4f51391eb01db
I20260621 05:16:57.474174 11148 ts_tablet_manager.cc:1939] T e0d556b4cd854560b9a4f51391eb01db P 03e0c2c807474eedbde39803bacbe51f: Deleting consensus metadata
I20260621 05:16:57.474520 10634 catalog_manager.cc:4841] TS 03e0c2c807474eedbde39803bacbe51f (127.0.98.3:36105): tablet e0d556b4cd854560b9a4f51391eb01db (table test-table [id=4b526a4538724777b337288e9a33b2ed]) successfully deleted
I20260621 05:16:57.478636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 10980
I20260621 05:16:57.483393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 10618
2026-06-21T05:16:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2008 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:16:57 dist-test-slave-czm5 krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:16:57 dist-test-slave-czm5 krb5kdc[11152](info): set up 2 sockets
Jun 21 05:16:57 dist-test-slave-czm5 krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260621 05:16:58.451591 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:16:58 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:16:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260621 05:16:58.595907 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaWoYkf/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/postgres -l logfile start
2026-06-21 05:17:00.776 UTC [11182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 05:17:00.776 UTC [11182] LOG: listening on IPv4 address "127.0.98.20", port 45199
2026-06-21 05:17:00.779 UTC [11182] LOG: listening on Unix socket "/tmp/.s.PGSQL.45199"
2026-06-21 05:17:00.811 UTC [11185] LOG: database system was shut down at 2026-06-21 05:16:59 UTC
2026-06-21 05:17:00.815 UTC [11182] LOG: database system is ready to accept connections
I20260621 05:17:01.715519 392 mini_postgres.cc:96] Postgres bound to 45199
2026-06-21 05:17:01.721 UTC [11193] FATAL: database "slave" does not exist
127.0.98.20:45199 - accepting connections
I20260621 05:17:01.721624 392 mini_ranger.cc:161] Starting Ranger...
I20260621 05:17:01.735435 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 05:17:02.137029 392 mini_ranger.cc:87] Created ranger Postgres database
I20260621 05:17:02.137121 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-admin
2026-06-21 05:17:02,387 [I] DB FLAVOR :POSTGRES
2026-06-21 05:17:02,388 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:17:02,388 [I] Checking connection..
2026-06-21 05:17:02,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 05:17:02,612 [I] Checking connection passed.
2026-06-21 05:17:02,612 [I] --------- Verifying version history table ---------
2026-06-21 05:17:02,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:17:02,837 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:17:02,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:17:03,064 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:17:03,064 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 05:17:03,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 05:17:03.302 UTC [11257] WARNING: there is no transaction in progress
2026-06-21 05:17:03,308 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 05:17:03,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:17:03,543 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 05:17:03,543 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:17:03,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 05:17:03,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 05:17:03,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-czm5.c.gcp-upstream.internal','N') ;"
2026-06-21 05:17:04,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:17:04,391 [I] Table x_portal_user does not exist in database ranger
2026-06-21 05:17:04,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:17:04,611 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 05:17:04,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:17:04,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:17:05,058 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 05:17:05,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 05:17:05.937 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:05.952 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:05.966 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:05.980 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:05.996 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.012 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.064 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.071 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.083 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.092 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.102 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.114 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.122 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.128 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.134 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06.139 UTC [11365] WARNING: there is no transaction in progress
2026-06-21 05:17:06,699 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 05:17:06,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 05:17:06,933 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 05:17:06,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:17:07,173 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 05:17:07,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:17:07,401 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 05:17:07,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:17:07,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:17:07,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-czm5.c.gcp-upstream.internal';"
2026-06-21 05:17:08,085 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 05:17:08,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 05:17:08,330 [I] DB_PATCHES have already been applied
I20260621 05:17:08.335950 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaWoYkf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 05:17:08.336010 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260621 05:17:08.338891 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50245
Jun 21, 2026 5:17:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:17:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:17:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 5:17:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 5:17:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 5:17:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 5:17:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 05:17:09.397225 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.676s user 0.000s sys 0.008s
Jun 21, 2026 5:17:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 5:17:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 21, 2026 5:17:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 5:17:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 5:17:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 05:17:28.475032 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 05:17:28 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019048, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 05:17:28.545689 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 05:17:28.553618 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 05:17:29.302943 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 05:17:29.303018 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms
I20260621 05:17:29.303035 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45199
2026-06-21 05:17:29,474 [I] DB FLAVOR :POSTGRES
2026-06-21 05:17:29,474 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:17:29,474 [I] Checking connection
2026-06-21 05:17:29,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 05:17:29,711 [I] connection success
2026-06-21 05:17:29,711 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 05:17:29,711 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 05:17:29,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 05:17:29,981 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 05:17:29,981 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:17:29,982 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 05:17:29,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 05:17:30,239 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 05:17:30.684587 392 mini_ranger_kms.cc:310] Created kms service
I20260621 05:17:30.878382 392 mini_ranger_kms.cc:326] Created kudu user
I20260621 05:17:30.916289 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 05:17:30 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019048, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:17:31.276566 392 mini_ranger_kms.cc:384] Added ranger policy
I20260621 05:17:31.276682 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 05:17:31.276703 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260621 05:17:31.279944 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11545
I20260621 05:17:31.280055 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56103
Jun 21, 2026 5:17:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:17:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:17:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 5:17:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 5:17:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 5:17:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 05:17:33.425272 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.880s user 0.005s sys 0.149s
I20260621 05:17:33.425372 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 05:17:33.425448 392 mini_ranger_kms.cc:401] 127.0.98.20:56103/kms/v1/keys
Jun 21 05:17:33 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 05:17:33 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 21 05:17:34 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 21 05:17:36 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:17:36.310566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44791
--ranger_config_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56103/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:17:36.386251 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:17:36.386436 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:17:36.386456 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:17:36.387576 11596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:17:36.387637 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:17:36.387701 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:17:36.387717 11596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:17:36.387744 11596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:17:36.389199 11596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56103/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44791
--ranger_config_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:17:36.389364 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:17:36.389590 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:17:36.393965 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:17:36.393947 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:17:36.394166 11596 server_base.cc:1034] running on GCE node
W20260621 05:17:36.393949 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:17:36.394465 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:17:36.394734 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:17:36.395887 11596 hybrid_clock.cc:648] HybridClock initialized: now 1782019056395870 us; error 30 us; skew 500 ppm
Jun 21 05:17:36 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:17:36.401785 11596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:17:36.402206 11596 webserver.cc:460] Webserver started at http://127.0.98.62:33847/ using document root <none> and password file <none>
I20260621 05:17:36.402415 11596 fs_manager.cc:359] Metadata directory not provided
I20260621 05:17:36.402472 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:17:36.402616 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:17:36 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:17:36.456811 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "b78f1d32724a43feadee8a6e45e89860"
format_stamp: "Formatted at 2026-06-21 05:17:36 on dist-test-slave-czm5"
server_key: "fb0c34cd49f6205ad3fa3cb23c357d0f"
server_key_iv: "69ffa1c5edc00d42d4ef27ffb9eb8677"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:36.457274 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "b78f1d32724a43feadee8a6e45e89860"
format_stamp: "Formatted at 2026-06-21 05:17:36 on dist-test-slave-czm5"
server_key: "fb0c34cd49f6205ad3fa3cb23c357d0f"
server_key_iv: "69ffa1c5edc00d42d4ef27ffb9eb8677"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:36.458825 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:17:36.472434 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:36.472595 11596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:17:36.472710 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "b78f1d32724a43feadee8a6e45e89860"
format_stamp: "Formatted at 2026-06-21 05:17:36 on dist-test-slave-czm5"
server_key: "fb0c34cd49f6205ad3fa3cb23c357d0f"
server_key_iv: "69ffa1c5edc00d42d4ef27ffb9eb8677"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:36.472786 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:17:36.491575 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:17:36.493304 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:17:36.493455 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:17:36.497714 11596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44791
I20260621 05:17:36.497764 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44791 every 8 connection(s)
I20260621 05:17:36.498106 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:17:36.498684 11664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:17:36.500720 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860: Bootstrap starting.
I20260621 05:17:36.501206 11664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860: Neither blocks nor log segments found. Creating new log.
I20260621 05:17:36.501410 11664 log.cc:824] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860: Log is configured to *not* fsync() on all Append() calls
I20260621 05:17:36.502009 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860: No bootstrap required, opened a new log
I20260621 05:17:36.503826 11664 raft_consensus.cc:348] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78f1d32724a43feadee8a6e45e89860" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44791 } }
I20260621 05:17:36.503948 11664 raft_consensus.cc:374] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:17:36.503971 11664 raft_consensus.cc:729] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78f1d32724a43feadee8a6e45e89860, State: Initialized, Role: FOLLOWER
I20260621 05:17:36.504086 11664 consensus_queue.cc:260] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78f1d32724a43feadee8a6e45e89860" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44791 } }
I20260621 05:17:36.504154 11664 raft_consensus.cc:388] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:17:36.504174 11664 raft_consensus.cc:482] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:17:36.504201 11664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:17:36.504657 11664 raft_consensus.cc:504] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78f1d32724a43feadee8a6e45e89860" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44791 } }
I20260621 05:17:36.504734 11664 leader_election.cc:302] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b78f1d32724a43feadee8a6e45e89860; no voters:
I20260621 05:17:36.504878 11664 leader_election.cc:288] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:17:36.504899 11667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:17:36.505017 11667 raft_consensus.cc:686] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [term 1 LEADER]: Becoming Leader. State: Replica: b78f1d32724a43feadee8a6e45e89860, State: Running, Role: LEADER
I20260621 05:17:36.505119 11664 sys_catalog.cc:564] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:17:36.505159 11667 consensus_queue.cc:237] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78f1d32724a43feadee8a6e45e89860" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44791 } }
I20260621 05:17:36.505517 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b78f1d32724a43feadee8a6e45e89860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78f1d32724a43feadee8a6e45e89860" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44791 } } }
I20260621 05:17:36.505542 11669 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b78f1d32724a43feadee8a6e45e89860. Latest consensus state: current_term: 1 leader_uuid: "b78f1d32724a43feadee8a6e45e89860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78f1d32724a43feadee8a6e45e89860" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44791 } } }
I20260621 05:17:36.505671 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [sys.catalog]: This master's current role is: LEADER
I20260621 05:17:36.505681 11669 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860 [sys.catalog]: This master's current role is: LEADER
I20260621 05:17:36.506054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 11596
I20260621 05:17:36.506181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:17:36.506386 392 external_mini_cluster.cc:1428] Setting key d1261ee763dc0a70f9d01698161f5725
I20260621 05:17:36.506642 11674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:17:36.506810 11664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 05:17:36.507115 11674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:17:36 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:17:36.518338 11674 catalog_manager.cc:1347] Generated new cluster ID: 91c9a00485bc4922938d946130d01dfa
I20260621 05:17:36.518392 11674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:17:36.520306 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:36.507104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.0.98.62:44791)
0621 05:17:36.509264 (+ 2160us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:36.509295 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:36.509779 (+ 484us) server_negotiation.cc:407] Connection header received
0621 05:17:36.510069 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:36.510080 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:36.510130 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:36.510298 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:17:36.510968 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:36.512434 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:36.513359 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:36.513780 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:36.513891 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:36.516043 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:17:36.516051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:17:36.516052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:17:36.516061 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:17:36.518271 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:36.518719 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:36.518722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:36.518722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:36.518818 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:36.519023 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:36.519024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:36.519024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:36.519337 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:17:36.519864 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:36.519935 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 05:17:36.520050 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2047,"thread_start_us":1354,"threads_started":1}
I20260621 05:17:36.547772 11674 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:17:36.549039 11674 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:17:36.555212 11674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860: Generated new TSK 0
I20260621 05:17:36.555446 11674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:17:37.162657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44791
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:17:37.279574 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:17:37.279871 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:17:37.279907 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:17:37.281368 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:17:37.281431 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:17:37.281467 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:17:37.283320 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:17:37.283540 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:17:37.283835 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:17:37.289605 11715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:17:37.289613 11714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:17:37.289901 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:17:37.290551 11709 server_base.cc:1034] running on GCE node
I20260621 05:17:37.290748 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:17:37.291003 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:17:37.292169 11709 hybrid_clock.cc:648] HybridClock initialized: now 1782019057292154 us; error 35 us; skew 500 ppm
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:17:37.294763 11709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:17:37.295161 11709 webserver.cc:460] Webserver started at http://127.0.98.1:41955/ using document root <none> and password file <none>
I20260621 05:17:37.295398 11709 fs_manager.cc:359] Metadata directory not provided
I20260621 05:17:37.295472 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:17:37.295612 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:17:37.308840 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "213ac2c80f614b04ad434c5b2754f84a"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "0799989a961a30e36be109a0c92139a4"
server_key_iv: "8102c3e3e117d837c2e777b35ffb2301"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.309263 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "213ac2c80f614b04ad434c5b2754f84a"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "0799989a961a30e36be109a0c92139a4"
server_key_iv: "8102c3e3e117d837c2e777b35ffb2301"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.311929 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 05:17:37.324404 11724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.324630 11709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:17:37.324725 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "213ac2c80f614b04ad434c5b2754f84a"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "0799989a961a30e36be109a0c92139a4"
server_key_iv: "8102c3e3e117d837c2e777b35ffb2301"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.324813 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:17:37.347012 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:17:37.349351 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:17:37.349542 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:17:37.349807 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:17:37.350178 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:17:37.350255 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.350340 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:17:37.350405 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.356892 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39917
I20260621 05:17:37.357185 11839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39917 every 8 connection(s)
I20260621 05:17:37.357798 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:17:37.359141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 11709
I20260621 05:17:37.359230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:17:37.359427 392 external_mini_cluster.cc:1428] Setting key 2db3b2b0bc301ac941cb238ae30b138e
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:17:37.380647 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.358900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39335 (local address 127.0.98.62:44791)
0621 05:17:37.369568 (+ 10668us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:37.369570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:37.369580 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:17:37.369604 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:37.369605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:37.369616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:37.370384 (+ 768us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:17:37.370440 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.371197 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.373135 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.373461 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.373539 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.377207 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:17:37.377209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:17:37.377209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:17:37.377217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:17:37.378993 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:37.379504 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:37.379505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:37.379506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:37.379572 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:37.379891 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:37.379892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:37.379893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:37.380050 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:17:37.380096 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:37.380467 (+ 371us) server_negotiation.cc:299] Negotiation successful
0621 05:17:37.380551 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10641,"thread_start_us":84,"threads_started":1}
I20260621 05:17:37.380856 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.358857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44791 (local address 127.0.98.1:39335)
0621 05:17:37.360512 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:37.360547 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:37.360936 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:37.369799 (+ 8863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:37.369804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:37.370011 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:17:37.370278 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.370285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.371311 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.371312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.373030 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.373033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.373569 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.373571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.373732 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.374415 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:17:37.374424 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:17:37.377102 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:17:37.379092 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:17:37.379099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:17:37.379101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:17:37.379389 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:17:37.379659 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:17:37.379660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:17:37.379661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:17:37.379804 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:17:37.380135 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:17:37.380141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:17:37.380302 (+ 161us) client_negotiation.cc:769] Sending connection context
0621 05:17:37.380427 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 05:17:37.380569 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1621,"thread_start_us":68,"threads_started":1}
I20260621 05:17:37.381330 11840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44791
I20260621 05:17:37.381441 11840 heartbeater.cc:464] Registering TS with master...
I20260621 05:17:37.381629 11840 heartbeater.cc:511] Master 127.0.98.62:44791 requested a full tablet report, sending...
I20260621 05:17:37.382153 11628 ts_manager.cc:194] Registered new tserver with Master: 213ac2c80f614b04ad434c5b2754f84a (127.0.98.1:39917)
I20260621 05:17:37.383015 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:17:37.392480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44791
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:17:37.504660 11846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:17:37.504904 11846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:17:37.504958 11846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:17:37.506498 11846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:17:37.506557 11846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:17:37.506595 11846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:17:37.508605 11846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:17:37.508821 11846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:17:37.509100 11846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 05:17:37.514505 11853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:17:37.514616 11854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:17:37.514430 11856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:17:37.514979 11846 server_base.cc:1034] running on GCE node
I20260621 05:17:37.515179 11846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:17:37.515445 11846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:17:37.516636 11846 hybrid_clock.cc:648] HybridClock initialized: now 1782019057516614 us; error 31 us; skew 500 ppm
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:17:37.519395 11846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:17:37.519898 11846 webserver.cc:460] Webserver started at http://127.0.98.2:37737/ using document root <none> and password file <none>
I20260621 05:17:37.520155 11846 fs_manager.cc:359] Metadata directory not provided
I20260621 05:17:37.520272 11846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:17:37.520448 11846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:17:37.538909 11846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "122291627fa042a2a6aa75e8fb34d3ba"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "b605b5ea18171ff8ea0e75c184767a4c"
server_key_iv: "8d3e9846b81b45a35a0591fa21f81219"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.539382 11846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "122291627fa042a2a6aa75e8fb34d3ba"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "b605b5ea18171ff8ea0e75c184767a4c"
server_key_iv: "8d3e9846b81b45a35a0591fa21f81219"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.541103 11846 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260621 05:17:37.557343 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.563752 11846 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.001s
I20260621 05:17:37.563874 11846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "122291627fa042a2a6aa75e8fb34d3ba"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "b605b5ea18171ff8ea0e75c184767a4c"
server_key_iv: "8d3e9846b81b45a35a0591fa21f81219"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.563966 11846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:17:37.586067 11846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:17:37.588945 11846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:17:37.589136 11846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:17:37.589407 11846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:17:37.589761 11846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:17:37.589846 11846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.589911 11846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:17:37.589963 11846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.597261 11846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43305
I20260621 05:17:37.597700 11846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:17:37.598495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 11846
I20260621 05:17:37.598588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:17:37.598747 392 external_mini_cluster.cc:1428] Setting key 9c2f9fc0323d35d2c0245febae5c5066
I20260621 05:17:37.601279 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43305 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:17:37.620792 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.601948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44791 (local address 127.0.98.2:33663)
0621 05:17:37.609181 (+ 7233us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:37.609227 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:37.609836 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:37.610166 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:37.610170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:37.610378 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:17:37.610628 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.610633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.611573 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.611574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.612785 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.612787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.613285 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.613286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.613414 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.614047 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:17:37.614057 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:17:37.617199 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:17:37.619119 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:17:37.619126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:17:37.619128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:17:37.619389 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:17:37.619593 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:17:37.619594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:17:37.619594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:17:37.619747 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:17:37.620079 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:17:37.620085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:17:37.620282 (+ 197us) client_negotiation.cc:769] Sending connection context
0621 05:17:37.620391 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 05:17:37.620546 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7186,"thread_start_us":80,"threads_started":1}
I20260621 05:17:37.621521 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.602212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33663 (local address 127.0.98.62:44791)
0621 05:17:37.602269 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:37.602271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:37.609313 (+ 7042us) server_negotiation.cc:407] Connection header received
0621 05:17:37.609954 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:37.609956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:37.609968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:37.609996 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:17:37.610728 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.611470 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.612888 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.613164 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.613203 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.617326 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:17:37.617328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:17:37.617328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:17:37.617345 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:17:37.619031 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:37.619474 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:37.619475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:37.619475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:37.619530 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:37.619851 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:37.619852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:37.619853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:37.619990 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:17:37.620051 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:37.621319 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0621 05:17:37.621426 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:17:37.622077 11978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44791
I20260621 05:17:37.622174 11978 heartbeater.cc:464] Registering TS with master...
I20260621 05:17:37.622362 11978 heartbeater.cc:511] Master 127.0.98.62:44791 requested a full tablet report, sending...
I20260621 05:17:37.622706 11628 ts_manager.cc:194] Registered new tserver with Master: 122291627fa042a2a6aa75e8fb34d3ba (127.0.98.2:43305)
I20260621 05:17:37.623178 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:17:37.632197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44791
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:17:37.744958 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:17:37.745316 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:17:37.745428 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:17:37.747087 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:17:37.747192 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:17:37.747272 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:17:37.749444 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:17:37.749888 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:17:37.750180 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:17:37.760636 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:17:37.760742 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:17:37.760946 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:17:37.764344 11983 server_base.cc:1034] running on GCE node
I20260621 05:17:37.764645 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:17:37.764894 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:17:37.766047 11983 hybrid_clock.cc:648] HybridClock initialized: now 1782019057766025 us; error 36 us; skew 500 ppm
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:17:37.783762 11983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:17:37.784292 11983 webserver.cc:460] Webserver started at http://127.0.98.3:34745/ using document root <none> and password file <none>
I20260621 05:17:37.784549 11983 fs_manager.cc:359] Metadata directory not provided
I20260621 05:17:37.784605 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:17:37.784729 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:17:37.806360 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "4cf7d0106a974d6e9f9717a8ee9768b1"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "0a1c57f037f2d658b044b20780507dab"
server_key_iv: "1ddfe145102b9ffff2c995488ea52418"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.806911 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4cf7d0106a974d6e9f9717a8ee9768b1"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "0a1c57f037f2d658b044b20780507dab"
server_key_iv: "1ddfe145102b9ffff2c995488ea52418"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.811892 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260621 05:17:37.829496 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.830093 11983 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260621 05:17:37.830246 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "4cf7d0106a974d6e9f9717a8ee9768b1"
format_stamp: "Formatted at 2026-06-21 05:17:37 on dist-test-slave-czm5"
server_key: "0a1c57f037f2d658b044b20780507dab"
server_key_iv: "1ddfe145102b9ffff2c995488ea52418"
server_key_version: "kuduclusterkey@0"
I20260621 05:17:37.830436 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:17:37.846773 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:17:37.849128 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:17:37.849277 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:17:37.849506 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:17:37.849885 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:17:37.849934 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.849965 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:17:37.850035 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:17:37.858603 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36799
I20260621 05:17:37.858741 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36799 every 8 connection(s)
I20260621 05:17:37.859082 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:17:37.872462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 11983
I20260621 05:17:37.872560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:17:37.872735 392 external_mini_cluster.cc:1428] Setting key 20367dda1dd8fc729a6e982daa7a5781
Jun 21 05:17:37 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:17:37.883774 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.860693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46819 (local address 127.0.98.62:44791)
0621 05:17:37.860799 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:37.860801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:37.864181 (+ 3380us) server_negotiation.cc:407] Connection header received
0621 05:17:37.864898 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:37.864900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:37.864913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:37.864944 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:17:37.865772 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.866608 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.868100 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.868501 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.868559 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.878846 (+ 10287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:17:37.878847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:17:37.878847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:17:37.878855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:17:37.880490 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:37.882666 (+ 2176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:37.882667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:37.882668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:37.882726 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:37.883018 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:37.883019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:37.883020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:37.883159 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:17:37.883205 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:37.883587 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 05:17:37.883683 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 05:17:37.885161 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.860418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44791 (local address 127.0.98.3:46819)
0621 05:17:37.864025 (+ 3607us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:37.864076 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:37.864750 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:37.865133 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:37.865137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:37.865398 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:17:37.865666 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.865672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.866719 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.866720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.868005 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.868008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.868657 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.868658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.868795 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.874922 (+ 6127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:17:37.874932 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:17:37.878758 (+ 3826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:17:37.881484 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:17:37.881490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:17:37.881493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:17:37.881784 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:17:37.882806 (+ 1022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:17:37.882807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:17:37.882807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:17:37.882943 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:17:37.883233 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:17:37.883238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:17:37.883428 (+ 190us) client_negotiation.cc:769] Sending connection context
0621 05:17:37.884754 (+ 1326us) client_negotiation.cc:240] Negotiation successful
0621 05:17:37.884934 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3559,"thread_start_us":70,"threads_started":1}
I20260621 05:17:37.885746 12112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44791
I20260621 05:17:37.885851 12112 heartbeater.cc:464] Registering TS with master...
I20260621 05:17:37.886037 12112 heartbeater.cc:511] Master 127.0.98.62:44791 requested a full tablet report, sending...
I20260621 05:17:37.886593 11628 ts_manager.cc:194] Registered new tserver with Master: 4cf7d0106a974d6e9f9717a8ee9768b1 (127.0.98.3:36799)
I20260621 05:17:37.887105 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46819
I20260621 05:17:37.895344 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:17:37.915807 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.900845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.98.62:44791)
0621 05:17:37.900888 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:37.900890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:37.901492 (+ 602us) server_negotiation.cc:407] Connection header received
0621 05:17:37.901594 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:37.901595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:37.901605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:37.901635 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:17:37.902474 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.903367 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.905136 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.905469 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.905511 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.911531 (+ 6020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:17:37.911533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:17:37.911533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:17:37.911541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:17:37.913349 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:37.914867 (+ 1518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:37.914869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:37.914869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:37.914929 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:17:37.915156 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:17:37.915157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:17:37.915157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:17:37.915283 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:17:37.915384 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:37.915607 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 05:17:37.915705 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:17:37.918211 11628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:17:37.920791 11628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:17:37.941133 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.927471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36799 (local address 127.0.0.1:50520)
0621 05:17:37.930068 (+ 2597us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:37.930110 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:37.930173 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:37.930987 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:37.930990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:37.930991 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:17:37.931059 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.931060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.934518 (+ 3458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.934519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.936419 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.936422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.940833 (+ 4411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.940835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.940935 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.940949 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:17:37.941012 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:17:37.941056 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2534,"thread_start_us":82,"threads_started":1}
I20260621 05:17:37.945788 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.928644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43305 (local address 127.0.0.1:34190)
0621 05:17:37.931696 (+ 3052us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:37.931706 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:37.931753 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:37.932732 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:37.932733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:37.932735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:17:37.932777 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.932778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.933944 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.933945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.942328 (+ 8383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.942331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.945550 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.945551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.945611 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.945616 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:17:37.945669 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:17:37.945709 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3037,"thread_start_us":72,"threads_started":1}
I20260621 05:17:37.946182 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.928257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.98.2:43305)
0621 05:17:37.930822 (+ 2565us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:37.930826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:37.931806 (+ 980us) server_negotiation.cc:407] Connection header received
0621 05:17:37.931844 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:37.931846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:37.931858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:37.931925 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:17:37.932845 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.933846 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.942442 (+ 8596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.944813 (+ 2371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.945429 (+ 616us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.945515 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:37.946058 (+ 543us) server_negotiation.cc:299] Negotiation successful
0621 05:17:37.946099 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2534,"thread_start_us":72,"threads_started":1}
I20260621 05:17:37.947149 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.928089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.0.98.3:36799)
0621 05:17:37.930569 (+ 2480us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:37.930574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:37.930583 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:17:37.930619 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:37.930622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:37.930637 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:37.930703 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:17:37.931128 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.934409 (+ 3281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.937527 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.938261 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.938330 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.946454 (+ 8124us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:37.946484 (+ 30us) server_negotiation.cc:299] Negotiation successful
0621 05:17:37.946532 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2433,"thread_start_us":68,"threads_started":1}
I20260621 05:17:37.948092 11912 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0c9b12ce894b87bec125691e712137 (DEFAULT_TABLE table=test-table [id=8d633f1ce91b45b1860adea863f6b682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:17:37.948457 11912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0c9b12ce894b87bec125691e712137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:17:37.956175 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0c9b12ce894b87bec125691e712137 (DEFAULT_TABLE table=test-table [id=8d633f1ce91b45b1860adea863f6b682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:17:37.956882 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0c9b12ce894b87bec125691e712137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:17:37.965665 12132 tablet_bootstrap.cc:492] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: Bootstrap starting.
I20260621 05:17:37.967006 12131 tablet_bootstrap.cc:492] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: Bootstrap starting.
I20260621 05:17:37.969115 12131 tablet_bootstrap.cc:654] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: Neither blocks nor log segments found. Creating new log.
I20260621 05:17:37.969313 12132 tablet_bootstrap.cc:654] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: Neither blocks nor log segments found. Creating new log.
I20260621 05:17:37.969648 12132 log.cc:824] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: Log is configured to *not* fsync() on all Append() calls
I20260621 05:17:37.970057 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.927785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39917 (local address 127.0.0.1:53480)
0621 05:17:37.930866 (+ 3081us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:37.930875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:37.930928 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:37.961386 (+ 30458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:37.961387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:37.961390 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:17:37.961444 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.961447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.963729 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.963730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.964774 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:37.964777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.969815 (+ 5038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.969816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:37.969882 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.969887 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:17:37.969932 (+ 45us) client_negotiation.cc:240] Negotiation successful
0621 05:17:37.969977 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3067,"thread_start_us":62,"threads_started":1}
I20260621 05:17:37.970641 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:37.927896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.0.98.1:39917)
0621 05:17:37.961208 (+ 33312us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:37.961213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:37.961225 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:17:37.961260 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:37.961261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:37.961278 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:37.961570 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:17:37.961641 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.963609 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.964886 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:37.968227 (+ 3341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:37.970426 (+ 2199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:37.970496 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:37.970524 (+ 28us) server_negotiation.cc:299] Negotiation successful
0621 05:17:37.970567 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33273,"thread_start_us":58,"threads_started":1}
I20260621 05:17:37.972215 11774 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0c9b12ce894b87bec125691e712137 (DEFAULT_TABLE table=test-table [id=8d633f1ce91b45b1860adea863f6b682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:17:37.972532 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0c9b12ce894b87bec125691e712137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:17:37.980027 12131 log.cc:824] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: Log is configured to *not* fsync() on all Append() calls
I20260621 05:17:37.980399 12134 tablet_bootstrap.cc:492] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a: Bootstrap starting.
I20260621 05:17:37.981043 12134 tablet_bootstrap.cc:654] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a: Neither blocks nor log segments found. Creating new log.
I20260621 05:17:37.981376 12134 log.cc:824] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a: Log is configured to *not* fsync() on all Append() calls
I20260621 05:17:37.982117 12132 tablet_bootstrap.cc:492] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: No bootstrap required, opened a new log
I20260621 05:17:37.982226 12132 ts_tablet_manager.cc:1397] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260621 05:17:37.984566 12132 raft_consensus.cc:348] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:37.984855 12132 raft_consensus.cc:374] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:17:37.984936 12132 raft_consensus.cc:729] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf7d0106a974d6e9f9717a8ee9768b1, State: Initialized, Role: FOLLOWER
I20260621 05:17:37.985122 12132 consensus_queue.cc:260] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:37.985495 12132 ts_tablet_manager.cc:1428] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:17:37.985651 12112 heartbeater.cc:503] Master 127.0.98.62:44791 was elected leader, sending a full tablet report...
I20260621 05:17:37.987107 12131 tablet_bootstrap.cc:492] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: No bootstrap required, opened a new log
I20260621 05:17:37.987224 12131 ts_tablet_manager.cc:1397] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20260621 05:17:37.989601 12131 raft_consensus.cc:348] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:37.989843 12131 raft_consensus.cc:374] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:17:37.989920 12131 raft_consensus.cc:729] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122291627fa042a2a6aa75e8fb34d3ba, State: Initialized, Role: FOLLOWER
I20260621 05:17:37.990099 12131 consensus_queue.cc:260] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:37.990434 12131 ts_tablet_manager.cc:1428] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:17:37.990571 11978 heartbeater.cc:503] Master 127.0.98.62:44791 was elected leader, sending a full tablet report...
I20260621 05:17:37.991743 12134 tablet_bootstrap.cc:492] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a: No bootstrap required, opened a new log
I20260621 05:17:37.991856 12134 ts_tablet_manager.cc:1397] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260621 05:17:37.994282 12134 raft_consensus.cc:348] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:37.994499 12134 raft_consensus.cc:374] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:17:37.994570 12134 raft_consensus.cc:729] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ac2c80f614b04ad434c5b2754f84a, State: Initialized, Role: FOLLOWER
I20260621 05:17:37.994738 12134 consensus_queue.cc:260] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:37.995007 12134 ts_tablet_manager.cc:1428] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:17:37.995155 11840 heartbeater.cc:503] Master 127.0.98.62:44791 was elected leader, sending a full tablet report...
I20260621 05:17:38.019711 12139 raft_consensus.cc:482] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:17:38.019949 12139 raft_consensus.cc:504] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:38.021178 12139 leader_election.cc:288] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf7d0106a974d6e9f9717a8ee9768b1 (127.0.98.3:36799), 122291627fa042a2a6aa75e8fb34d3ba (127.0.98.2:43305)
I20260621 05:17:38.024032 12137 raft_consensus.cc:482] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:17:38.024132 12137 raft_consensus.cc:504] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:38.024466 12137 leader_election.cc:288] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 122291627fa042a2a6aa75e8fb34d3ba (127.0.98.2:43305), 213ac2c80f614b04ad434c5b2754f84a (127.0.98.1:39917)
Jun 21 05:17:38 dist-test-slave-czm5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:17:38.035068 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.020764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39919 (local address 127.0.98.3:36799)
0621 05:17:38.020810 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:38.020812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:38.023011 (+ 2199us) server_negotiation.cc:407] Connection header received
0621 05:17:38.023438 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:38.023439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:38.023451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:38.023476 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:17:38.025477 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.026200 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.034002 (+ 7802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.034516 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.034577 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.034586 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:38.034944 (+ 358us) server_negotiation.cc:299] Negotiation successful
0621 05:17:38.034984 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:17:38.035532 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.020568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36799 (local address 127.0.98.1:39919)
0621 05:17:38.022937 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:38.022948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:38.023349 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:38.023539 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:38.023540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:38.023541 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:17:38.023636 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:38.023638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.026310 (+ 2672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.026310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:38.027585 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:38.027588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.034748 (+ 7160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.034749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:38.034815 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.034819 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:17:38.035384 (+ 565us) client_negotiation.cc:240] Negotiation successful
0621 05:17:38.035448 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2352,"thread_start_us":90,"threads_started":1}
I20260621 05:17:38.036024 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.021122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57841 (local address 127.0.98.2:43305)
0621 05:17:38.021160 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:38.021161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:38.028590 (+ 7429us) server_negotiation.cc:407] Connection header received
0621 05:17:38.028682 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:38.028683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:38.028691 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:38.028717 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:17:38.028906 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.029650 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.033382 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.033900 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.033967 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.033977 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:38.035905 (+ 1928us) server_negotiation.cc:299] Negotiation successful
0621 05:17:38.035947 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:17:38.036471 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.020946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43305 (local address 127.0.98.1:57841)
0621 05:17:38.028529 (+ 7583us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:38.028539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:38.028615 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:38.028773 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:38.028774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:38.028775 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:17:38.028836 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:38.028839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.029775 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.029776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:38.030809 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:38.030811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.034869 (+ 4058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.034870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:38.035785 (+ 915us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.035789 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:17:38.036307 (+ 518us) client_negotiation.cc:240] Negotiation successful
0621 05:17:38.036381 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7564,"thread_start_us":67,"threads_started":1}
I20260621 05:17:38.036769 12061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c9b12ce894b87bec125691e712137" candidate_uuid: "213ac2c80f614b04ad434c5b2754f84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" is_pre_election: true
I20260621 05:17:38.036908 12061 raft_consensus.cc:2454] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213ac2c80f614b04ad434c5b2754f84a in term 0.
I20260621 05:17:38.037042 11932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c9b12ce894b87bec125691e712137" candidate_uuid: "213ac2c80f614b04ad434c5b2754f84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122291627fa042a2a6aa75e8fb34d3ba" is_pre_election: true
I20260621 05:17:38.037132 11726 leader_election.cc:302] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213ac2c80f614b04ad434c5b2754f84a, 4cf7d0106a974d6e9f9717a8ee9768b1; no voters:
I20260621 05:17:38.037178 11932 raft_consensus.cc:2454] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213ac2c80f614b04ad434c5b2754f84a in term 0.
I20260621 05:17:38.037405 12139 raft_consensus.cc:2781] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:17:38.037472 12139 raft_consensus.cc:482] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:17:38.037498 12139 raft_consensus.cc:3037] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:17:38.038429 12139 raft_consensus.cc:504] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:38.038558 12139 leader_election.cc:288] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [CANDIDATE]: Term 1 election: Requested vote from peers 4cf7d0106a974d6e9f9717a8ee9768b1 (127.0.98.3:36799), 122291627fa042a2a6aa75e8fb34d3ba (127.0.98.2:43305)
I20260621 05:17:38.038825 11932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c9b12ce894b87bec125691e712137" candidate_uuid: "213ac2c80f614b04ad434c5b2754f84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122291627fa042a2a6aa75e8fb34d3ba"
I20260621 05:17:38.038905 11932 raft_consensus.cc:3037] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:17:38.039762 11932 raft_consensus.cc:2454] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213ac2c80f614b04ad434c5b2754f84a in term 1.
I20260621 05:17:38.039906 12061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c9b12ce894b87bec125691e712137" candidate_uuid: "213ac2c80f614b04ad434c5b2754f84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1"
I20260621 05:17:38.039969 12061 raft_consensus.cc:3037] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:17:38.040609 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.024645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43305 (local address 127.0.98.3:51351)
0621 05:17:38.025069 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:38.025080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:38.025122 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:38.033770 (+ 8648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:38.033771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:38.033773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:17:38.033833 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:38.033834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.038533 (+ 4699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.038534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:38.039680 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:38.039683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.040374 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.040374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:38.040449 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.040454 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:17:38.040482 (+ 28us) client_negotiation.cc:240] Negotiation successful
0621 05:17:38.040542 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409}
I20260621 05:17:38.040751 12061 raft_consensus.cc:2454] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213ac2c80f614b04ad434c5b2754f84a in term 1.
I20260621 05:17:38.040882 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.025243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51351 (local address 127.0.98.2:43305)
0621 05:17:38.033637 (+ 8394us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:38.033639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:38.033646 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:17:38.033668 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:38.033669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:38.033677 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:38.033900 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:17:38.033933 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.034613 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.039775 (+ 5162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.040283 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.040735 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.040749 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:38.040768 (+ 19us) server_negotiation.cc:299] Negotiation successful
0621 05:17:38.040805 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8369,"thread_start_us":93,"threads_started":1}
I20260621 05:17:38.041157 11726 leader_election.cc:302] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213ac2c80f614b04ad434c5b2754f84a, 4cf7d0106a974d6e9f9717a8ee9768b1; no voters:
I20260621 05:17:38.041298 12139 raft_consensus.cc:2781] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:17:38.041451 12139 raft_consensus.cc:686] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [term 1 LEADER]: Becoming Leader. State: Replica: 213ac2c80f614b04ad434c5b2754f84a, State: Running, Role: LEADER
I20260621 05:17:38.041452 11932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c9b12ce894b87bec125691e712137" candidate_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122291627fa042a2a6aa75e8fb34d3ba" is_pre_election: true
I20260621 05:17:38.041531 11932 raft_consensus.cc:2379] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cf7d0106a974d6e9f9717a8ee9768b1 in current term 1: Already voted for candidate 213ac2c80f614b04ad434c5b2754f84a in this term.
I20260621 05:17:38.041535 12139 consensus_queue.cc:237] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } }
I20260621 05:17:38.042347 11628 catalog_manager.cc:5495] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ac2c80f614b04ad434c5b2754f84a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "213ac2c80f614b04ad434c5b2754f84a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "213ac2c80f614b04ad434c5b2754f84a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39917 } health_report { overall_health: HEALTHY } } }
I20260621 05:17:38.043782 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.025386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51803 (local address 127.0.98.1:39917)
0621 05:17:38.025421 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:38.025423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:38.034603 (+ 9180us) server_negotiation.cc:407] Connection header received
0621 05:17:38.034625 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:38.034625 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:38.034633 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:38.034674 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:17:38.037606 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.038267 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.040799 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.041277 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.041320 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.041328 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:38.043699 (+ 2371us) server_negotiation.cc:299] Negotiation successful
0621 05:17:38.043737 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:17:38.044111 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.024753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39917 (local address 127.0.98.3:51803)
0621 05:17:38.033473 (+ 8720us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:38.033496 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 05:17:38.033543 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:17:38.037459 (+ 3916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:17:38.037459 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:17:38.037462 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:17:38.037517 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:38.037519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.038352 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.038353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:38.039500 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:17:38.039503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.041803 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.041803 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:17:38.041883 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.041886 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:17:38.043990 (+ 2104us) client_negotiation.cc:240] Negotiation successful
0621 05:17:38.044028 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8691,"thread_start_us":73,"threads_started":1}
I20260621 05:17:38.044521 11794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c9b12ce894b87bec125691e712137" candidate_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213ac2c80f614b04ad434c5b2754f84a" is_pre_election: true
I20260621 05:17:38.044842 11999 leader_election.cc:302] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cf7d0106a974d6e9f9717a8ee9768b1; no voters: 122291627fa042a2a6aa75e8fb34d3ba, 213ac2c80f614b04ad434c5b2754f84a
I20260621 05:17:38.044960 12137 raft_consensus.cc:2726] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 05:17:38.060211 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.057024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.0.98.1:39917)
0621 05:17:38.057061 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:17:38.057062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:17:38.057105 (+ 43us) server_negotiation.cc:407] Connection header received
0621 05:17:38.057182 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:17:38.057183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:17:38.057202 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:17:38.057269 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:17:38.057479 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.058066 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.058981 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:17:38.059256 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:17:38.059333 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:17:38.059521 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:17:38.059690 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:17:38.059758 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:17:38.060102 (+ 344us) server_negotiation.cc:299] Negotiation successful
0621 05:17:38.060123 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:17:38.068634 11932 raft_consensus.cc:1261] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 1 FOLLOWER]: Refusing update from remote peer 213ac2c80f614b04ad434c5b2754f84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:17:38.069010 12144 consensus_queue.cc:1035] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "122291627fa042a2a6aa75e8fb34d3ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:17:38.071089 12061 raft_consensus.cc:1261] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 1 FOLLOWER]: Refusing update from remote peer 213ac2c80f614b04ad434c5b2754f84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:17:38.071404 12144 consensus_queue.cc:1035] T 9c0c9b12ce894b87bec125691e712137 P 213ac2c80f614b04ad434c5b2754f84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf7d0106a974d6e9f9717a8ee9768b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:17:38.073971 12147 mvcc.cc:205] Tried to move back new op lower bound from 7299150061846081536 to 7299150061738655744. Current Snapshot: MvccSnapshot[applied={T|T < 7299150061846081536}]
I20260621 05:17:38.075225 12150 mvcc.cc:205] Tried to move back new op lower bound from 7299150061846081536 to 7299150061738655744. Current Snapshot: MvccSnapshot[applied={T|T < 7299150061846081536}]
I20260621 05:17:38.077329 12148 mvcc.cc:205] Tried to move back new op lower bound from 7299150061846081536 to 7299150061738655744. Current Snapshot: MvccSnapshot[applied={T|T < 7299150061846081536}]
I20260621 05:17:38.079928 11628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:17:38.080008 11628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:17:38.080693 11628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b78f1d32724a43feadee8a6e45e89860: Sending DeleteTablet for 3 replicas of tablet 9c0c9b12ce894b87bec125691e712137
I20260621 05:17:38.081156 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0c9b12ce894b87bec125691e712137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:17:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50520
I20260621 05:17:38.081382 11912 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0c9b12ce894b87bec125691e712137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:17:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34190
I20260621 05:17:38.081771 11774 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0c9b12ce894b87bec125691e712137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:17:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53480
I20260621 05:17:38.082250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 11709
I20260621 05:17:38.084697 12156 tablet_replica.cc:330] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: stopping tablet replica
I20260621 05:17:38.084846 12157 tablet_replica.cc:330] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: stopping tablet replica
I20260621 05:17:38.085026 12157 raft_consensus.cc:2229] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:17:38.085206 12157 raft_consensus.cc:2258] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:17:38.085356 12156 raft_consensus.cc:2229] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:17:38.085582 12156 raft_consensus.cc:2258] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:17:38.089715 12157 ts_tablet_manager.cc:1905] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:17:38.090114 12156 ts_tablet_manager.cc:1905] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:17:38.093078 12156 ts_tablet_manager.cc:1918] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:17:38.093679 12157 ts_tablet_manager.cc:1918] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:17:38.093924 12156 log.cc:1196] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/9c0c9b12ce894b87bec125691e712137
I20260621 05:17:38.094269 12156 ts_tablet_manager.cc:1939] T 9c0c9b12ce894b87bec125691e712137 P 4cf7d0106a974d6e9f9717a8ee9768b1: Deleting consensus metadata
I20260621 05:17:38.094378 12157 log.cc:1196] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/9c0c9b12ce894b87bec125691e712137
I20260621 05:17:38.094830 11613 catalog_manager.cc:4841] TS 4cf7d0106a974d6e9f9717a8ee9768b1 (127.0.98.3:36799): tablet 9c0c9b12ce894b87bec125691e712137 (table test-table [id=8d633f1ce91b45b1860adea863f6b682]) successfully deleted
I20260621 05:17:38.095180 12157 ts_tablet_manager.cc:1939] T 9c0c9b12ce894b87bec125691e712137 P 122291627fa042a2a6aa75e8fb34d3ba: Deleting consensus metadata
I20260621 05:17:38.095654 11613 catalog_manager.cc:4841] TS 122291627fa042a2a6aa75e8fb34d3ba (127.0.98.2:43305): tablet 9c0c9b12ce894b87bec125691e712137 (table test-table [id=8d633f1ce91b45b1860adea863f6b682]) successfully deleted
W20260621 05:17:38.096109 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39917 (error 108)
I20260621 05:17:38.096891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 11846
I20260621 05:17:38.096912 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:17:38.096766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39917 (local address 127.0.0.1:53484)
0621 05:17:38.096816 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:17:38.096866 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260621 05:17:38.097734 11612 catalog_manager.cc:4584] TS 213ac2c80f614b04ad434c5b2754f84a (127.0.98.1:39917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c0c9b12ce894b87bec125691e712137: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39917: connect: Connection refused (error 111)
I20260621 05:17:38.111582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 11983
I20260621 05:17:38.118470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 11596
I20260621 05:17:38.127266 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 05:17:38.472324 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 05:17:38.472417 392 mini_ranger.cc:66] Stopping Ranger...
I20260621 05:17:38.817589 392 mini_ranger.cc:68] Stopped Ranger
2026-06-21 05:17:38.817 UTC [11182] LOG: received smart shutdown request
2026-06-21 05:17:38.821 UTC [11182] LOG: background worker "logical replication launcher" (PID 11191) exited with exit code 1
2026-06-21 05:17:38.821 UTC [11186] LOG: shutting down
2026-06-21 05:17:38.962 UTC [11182] LOG: database system is shut down
2026-06-21T05:17:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (41586 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:17:39 dist-test-slave-czm5 krb5kdc[12168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:17:39 dist-test-slave-czm5 krb5kdc[12168](info): set up 2 sockets
Jun 21 05:17:39 dist-test-slave-czm5 krb5kdc[12168](info): commencing operation
krb5kdc: starting...
W20260621 05:17:40.047721 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:17:40 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:17:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260621 05:17:40.191931 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaWoYkf/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/postgres -l logfile start
2026-06-21 05:17:42.400 UTC [12198] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 05:17:42.400 UTC [12198] LOG: listening on IPv4 address "127.0.98.20", port 53985
2026-06-21 05:17:42.403 UTC [12198] LOG: listening on Unix socket "/tmp/.s.PGSQL.53985"
2026-06-21 05:17:42.436 UTC [12201] LOG: database system was shut down at 2026-06-21 05:17:41 UTC
2026-06-21 05:17:42.440 UTC [12198] LOG: database system is ready to accept connections
I20260621 05:17:43.347419 392 mini_postgres.cc:96] Postgres bound to 53985
2026-06-21 05:17:43.353 UTC [12209] FATAL: database "slave" does not exist
127.0.98.20:53985 - accepting connections
I20260621 05:17:43.353688 392 mini_ranger.cc:161] Starting Ranger...
I20260621 05:17:43.367590 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 05:17:43.769181 392 mini_ranger.cc:87] Created ranger Postgres database
I20260621 05:17:43.769246 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-admin
2026-06-21 05:17:44,022 [I] DB FLAVOR :POSTGRES
2026-06-21 05:17:44,022 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:17:44,022 [I] Checking connection..
2026-06-21 05:17:44,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 05:17:44,251 [I] Checking connection passed.
2026-06-21 05:17:44,252 [I] --------- Verifying version history table ---------
2026-06-21 05:17:44,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:17:44,470 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:17:44,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:17:44,692 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:17:44,692 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 05:17:44,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 05:17:44.905 UTC [12273] WARNING: there is no transaction in progress
2026-06-21 05:17:44,910 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 05:17:44,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:17:45,146 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 05:17:45,147 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:17:45,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 05:17:45,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 05:17:45,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-czm5.c.gcp-upstream.internal','N') ;"
2026-06-21 05:17:45,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:17:46,076 [I] Table x_portal_user does not exist in database ranger
2026-06-21 05:17:46,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:17:46,318 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 05:17:46,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:17:46,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:17:46,801 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 05:17:46,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 05:17:47.670 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.684 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.698 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.711 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.725 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.741 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.789 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.795 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.807 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.816 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.825 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.837 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.846 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.852 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.863 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:47.868 UTC [12381] WARNING: there is no transaction in progress
2026-06-21 05:17:48,421 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 05:17:48,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 05:17:48,658 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 05:17:48,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:17:48,871 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 05:17:48,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:17:49,102 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 05:17:49,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:17:49,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:17:49,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-czm5.c.gcp-upstream.internal';"
2026-06-21 05:17:49,763 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 05:17:49,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 05:17:49,993 [I] DB_PATCHES have already been applied
I20260621 05:17:49.998488 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaWoYkf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 05:17:49.998572 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260621 05:17:50.001365 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:43447
Jun 21, 2026 5:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 5:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 5:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 5:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 5:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 05:17:51.053757 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.700s user 0.000s sys 0.008s
Jun 21, 2026 5:18:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 5:18:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 5:18:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 5:18:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 5:18:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 05:18:11.817545 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 05:18:11 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019091, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 05:18:11.892048 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 05:18:11.900751 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 05:18:12.684430 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 05:18:12.684499 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms
I20260621 05:18:12.684512 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53985
2026-06-21 05:18:12,845 [I] DB FLAVOR :POSTGRES
2026-06-21 05:18:12,845 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:18:12,845 [I] Checking connection
2026-06-21 05:18:12,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 05:18:13,082 [I] connection success
2026-06-21 05:18:13,083 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 05:18:13,083 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 05:18:13,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 05:18:13,306 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 05:18:13,306 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:18:13,306 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 05:18:13,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53985/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 05:18:13,538 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 05:18:13.984999 392 mini_ranger_kms.cc:310] Created kms service
I20260621 05:18:14.155505 392 mini_ranger_kms.cc:326] Created kudu user
I20260621 05:18:14.201109 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 05:18:14 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019091, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:14.603752 392 mini_ranger_kms.cc:384] Added ranger policy
I20260621 05:18:14.603859 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaWoYkf/build/release/bin/postgresql.jar:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaWoYkf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 05:18:14.603876 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260621 05:18:14.607154 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12561
I20260621 05:18:14.607230 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33727
Jun 21, 2026 5:18:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:18:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:18:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 5:18:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 5:18:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 5:18:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 05:18:16 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019096, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:16.844101 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.952s user 0.002s sys 0.141s
I20260621 05:18:16.844187 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 05:18:16.844228 392 mini_ranger_kms.cc:401] 127.0.98.20:33727/kms/v1/keys
Jun 21 05:18:17 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019096, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 21 05:18:17 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019096, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 21 05:18:19 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:18:19.614722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36909
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36909
--ranger_config_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33727/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:19.701689 12612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:19.701880 12612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:19.701900 12612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:19.702920 12612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:19.702955 12612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:19.702972 12612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:19.702986 12612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:19.702996 12612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:19.704526 12612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33727/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36909
--ranger_config_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:19.704697 12612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:19.704846 12612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:19.709416 12620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:19.709439 12617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:19.709635 12612 server_base.cc:1034] running on GCE node
W20260621 05:18:19.709455 12618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:19.709923 12612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:19.710197 12612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:19.711360 12612 hybrid_clock.cc:648] HybridClock initialized: now 1782019099711347 us; error 27 us; skew 500 ppm
Jun 21 05:18:19 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:19.713819 12612 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:18:19.714231 12612 webserver.cc:460] Webserver started at http://127.0.98.62:45433/ using document root <none> and password file <none>
I20260621 05:18:19.714447 12612 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:19.714510 12612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:19.714659 12612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:18:19 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:19.774819 12612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "bf0eaf386598445a90e11d209189a2de"
format_stamp: "Formatted at 2026-06-21 05:18:19 on dist-test-slave-czm5"
server_key: "075fac6dc0121ac409d47c0aba290710"
server_key_iv: "9a2ac50aa0978619087af38dafc40002"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:19.775262 12612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf0eaf386598445a90e11d209189a2de"
format_stamp: "Formatted at 2026-06-21 05:18:19 on dist-test-slave-czm5"
server_key: "075fac6dc0121ac409d47c0aba290710"
server_key_iv: "9a2ac50aa0978619087af38dafc40002"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:19.776826 12612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:18:19.791244 12627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:19.791524 12612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:19.791643 12612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "bf0eaf386598445a90e11d209189a2de"
format_stamp: "Formatted at 2026-06-21 05:18:19 on dist-test-slave-czm5"
server_key: "075fac6dc0121ac409d47c0aba290710"
server_key_iv: "9a2ac50aa0978619087af38dafc40002"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:19.791754 12612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:19.809257 12612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:19.811236 12612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:19.811394 12612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:19.816263 12612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36909
I20260621 05:18:19.816357 12679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36909 every 8 connection(s)
I20260621 05:18:19.816705 12612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:18:19.817377 12680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:19.819617 12680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Bootstrap starting.
I20260621 05:18:19.820268 12680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:19.820565 12680 log.cc:824] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:19.820935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 12612
I20260621 05:18:19.821030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:18:19.821210 392 external_mini_cluster.cc:1428] Setting key 2d758647ea3830ee23fe562090032d3a
I20260621 05:18:19.821239 12680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: No bootstrap required, opened a new log
I20260621 05:18:19.823518 12680 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } }
I20260621 05:18:19.823714 12680 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:19.823777 12680 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0eaf386598445a90e11d209189a2de, State: Initialized, Role: FOLLOWER
I20260621 05:18:19.823956 12680 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } }
I20260621 05:18:19.824059 12680 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:18:19.824092 12680 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:18:19.824167 12680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:19.824822 12680 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } }
I20260621 05:18:19.824939 12680 leader_election.cc:302] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0eaf386598445a90e11d209189a2de; no voters:
I20260621 05:18:19.825150 12680 leader_election.cc:288] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:18:19.825192 12685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:19.825400 12685 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 1 LEADER]: Becoming Leader. State: Replica: bf0eaf386598445a90e11d209189a2de, State: Running, Role: LEADER
I20260621 05:18:19.825491 12680 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:19.825529 12685 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } }
I20260621 05:18:19.825896 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf0eaf386598445a90e11d209189a2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } } }
I20260621 05:18:19.826175 12686 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0eaf386598445a90e11d209189a2de. Latest consensus state: current_term: 1 leader_uuid: "bf0eaf386598445a90e11d209189a2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } } }
I20260621 05:18:19.826333 12686 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:19.826576 12695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:18:19.827054 12695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:18:19.827322 12680 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 05:18:19.826049 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: This master's current role is: LEADER
Jun 21 05:18:19 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:19.833917 12695 catalog_manager.cc:1347] Generated new cluster ID: 7ae9a834866d406487b662952de7209e
I20260621 05:18:19.833981 12695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:18:19.839468 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:19.821752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.0.98.62:36909)
0621 05:18:19.821906 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:19.821916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:19.821951 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:18:19.823532 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:19.823541 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:19.823607 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:19.823771 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:19.824627 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:19.826038 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:19.827174 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:19.827809 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:19.827913 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:19.831173 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:19.831210 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:19.831210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:19.831250 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:19.836935 (+ 5685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:19.837386 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:19.837389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:19.837389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:19.837572 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:19.837808 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:19.837809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:19.837810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:19.838203 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:19.838377 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:19.838892 (+ 515us) server_negotiation.cc:299] Negotiation successful
0621 05:18:19.839192 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260621 05:18:19.859954 12695 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:18:19.860692 12695 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:18:19.866703 12695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Generated new TSK 0
I20260621 05:18:19.866952 12695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:18:20.524068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36909
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:20.638230 12725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:20.638486 12725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:20.638521 12725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:20.640134 12725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:20.640192 12725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:20.640230 12725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:18:20.642333 12725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:20.642542 12725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:20.642787 12725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:20.648046 12731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:20.648047 12733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:20.648448 12730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:20.648850 12725 server_base.cc:1034] running on GCE node
I20260621 05:18:20.649093 12725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:20.649322 12725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:20.650473 12725 hybrid_clock.cc:648] HybridClock initialized: now 1782019100650454 us; error 39 us; skew 500 ppm
Jun 21 05:18:20 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:20.653151 12725 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:18:20.653647 12725 webserver.cc:460] Webserver started at http://127.0.98.1:42403/ using document root <none> and password file <none>
I20260621 05:18:20.653898 12725 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:20.653968 12725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:20.654089 12725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:18:20 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:20.666014 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0b6a4d4d79d4bbca51a3be431b0646c"
format_stamp: "Formatted at 2026-06-21 05:18:20 on dist-test-slave-czm5"
server_key: "6d49a0d9d497e86dd8c01a83dd170d24"
server_key_iv: "e4fdbeeccb1b6679483ec9f8de82037c"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:20.666379 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0b6a4d4d79d4bbca51a3be431b0646c"
format_stamp: "Formatted at 2026-06-21 05:18:20 on dist-test-slave-czm5"
server_key: "6d49a0d9d497e86dd8c01a83dd170d24"
server_key_iv: "e4fdbeeccb1b6679483ec9f8de82037c"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:20.667814 12725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:18:20.680444 12741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:20.680707 12725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:20.680819 12725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "d0b6a4d4d79d4bbca51a3be431b0646c"
format_stamp: "Formatted at 2026-06-21 05:18:20 on dist-test-slave-czm5"
server_key: "6d49a0d9d497e86dd8c01a83dd170d24"
server_key_iv: "e4fdbeeccb1b6679483ec9f8de82037c"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:20.680918 12725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:20.704344 12725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:20.706787 12725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:20.706928 12725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:20.707151 12725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:20.707482 12725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:20.707530 12725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:20.707558 12725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:20.707576 12725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:20.713562 12725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43663
I20260621 05:18:20.713994 12725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:18:20.721637 12855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43663 every 8 connection(s)
I20260621 05:18:20.722424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 12725
I20260621 05:18:20.722506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:18:20.722677 392 external_mini_cluster.cc:1428] Setting key 47638af3febdc247f2ea30a9f73d270e
Jun 21 05:18:20 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:20.734097 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:20.722249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36909 (local address 127.0.98.1:51401)
0621 05:18:20.722786 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:20.723374 (+ 588us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:20.723995 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:20.724384 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:20.724388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:20.724619 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:20.724931 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:20.724937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:20.725978 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:20.725979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:20.727433 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:20.727436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:20.728014 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:20.728015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:20.728149 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:20.728750 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:20.728760 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:20.730545 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:20.732550 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:20.732556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:20.732558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:20.732846 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:20.733076 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:20.733076 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:20.733077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:20.733195 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:20.733465 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:20.733471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:20.733629 (+ 158us) client_negotiation.cc:769] Sending connection context
0621 05:18:20.733732 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 05:18:20.733879 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":86,"threads_started":1}
I20260621 05:18:20.734553 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:20.722531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51401 (local address 127.0.98.62:36909)
0621 05:18:20.724166 (+ 1635us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:20.724168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:20.724176 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:18:20.724197 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:20.724198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:20.724209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:20.724268 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:20.725044 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:20.725825 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:20.727522 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:20.727866 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:20.727939 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:20.730662 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:20.730664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:20.730664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:20.730671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:20.732449 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:20.732945 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:20.732946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:20.732947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:20.733008 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:20.733262 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:20.733263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:20.733263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:20.733383 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:20.733440 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:20.734395 (+ 955us) server_negotiation.cc:299] Negotiation successful
0621 05:18:20.734470 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1607,"thread_start_us":71,"threads_started":1}
I20260621 05:18:20.735208 12856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36909
I20260621 05:18:20.735317 12856 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:20.735478 12856 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:20.736018 12644 ts_manager.cc:194] Registered new tserver with Master: d0b6a4d4d79d4bbca51a3be431b0646c (127.0.98.1:43663)
I20260621 05:18:20.736621 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51401
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:18:20.761953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36909
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:20.859035 12862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:20.859247 12862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:20.859282 12862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:20.860714 12862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:20.860781 12862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:20.860827 12862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:18:20.862366 12862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:20.862617 12862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:20.862856 12862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:20.867704 12869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:20.867799 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:20.867704 12870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:20.868809 12862 server_base.cc:1034] running on GCE node
I20260621 05:18:20.869014 12862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:20.869287 12862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Jun 21 05:18:20 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:20.870462 12862 hybrid_clock.cc:648] HybridClock initialized: now 1782019100870456 us; error 30 us; skew 500 ppm
Jun 21 05:18:20 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:20.872884 12862 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:18:20.873342 12862 webserver.cc:460] Webserver started at http://127.0.98.2:34099/ using document root <none> and password file <none>
I20260621 05:18:20.873606 12862 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:20.873675 12862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:20.873803 12862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:18:20 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:20.885154 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "1ab829b1cf684fa39da559ec93e14aca"
format_stamp: "Formatted at 2026-06-21 05:18:20 on dist-test-slave-czm5"
server_key: "7e7bba600d3158aa71622e3ef645ef6e"
server_key_iv: "11f6a0abb4125e7435ecb43f40aeea45"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:20.885777 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1ab829b1cf684fa39da559ec93e14aca"
format_stamp: "Formatted at 2026-06-21 05:18:20 on dist-test-slave-czm5"
server_key: "7e7bba600d3158aa71622e3ef645ef6e"
server_key_iv: "11f6a0abb4125e7435ecb43f40aeea45"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:20.891177 12862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260621 05:18:20.910926 12879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:20.911156 12862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:20.911245 12862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "1ab829b1cf684fa39da559ec93e14aca"
format_stamp: "Formatted at 2026-06-21 05:18:20 on dist-test-slave-czm5"
server_key: "7e7bba600d3158aa71622e3ef645ef6e"
server_key_iv: "11f6a0abb4125e7435ecb43f40aeea45"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:20.911352 12862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:20.932180 12862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:20.934105 12862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:20.934250 12862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:20.934458 12862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:20.934827 12862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:20.934907 12862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:20.934974 12862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:20.935029 12862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:20.941721 12862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37065
I20260621 05:18:20.942164 12862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:18:20.943827 12992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37065 every 8 connection(s)
I20260621 05:18:20.950423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 12862
I20260621 05:18:20.950516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:18:20.950731 392 external_mini_cluster.cc:1428] Setting key 5451904a271b72805b480414dc6fc544
Jun 21 05:18:20 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:20.955384 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:20.944761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58883 (local address 127.0.98.62:36909)
0621 05:18:20.944841 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:20.944843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:20.945485 (+ 642us) server_negotiation.cc:407] Connection header received
0621 05:18:20.946157 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:20.946159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:20.946171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:20.946200 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:20.946966 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:20.947784 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:20.949202 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:20.949522 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:20.949561 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:20.952202 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:20.952203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:20.952204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:20.952210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:20.953875 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:20.954325 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:20.954327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:20.954327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:20.954384 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:20.954646 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:20.954647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:20.954647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:20.954787 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:20.954832 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:20.955194 (+ 362us) server_negotiation.cc:299] Negotiation successful
0621 05:18:20.955276 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:18:20.956107 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:20.944505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36909 (local address 127.0.98.2:58883)
0621 05:18:20.945367 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:20.945408 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:20.946037 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:20.946377 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:20.946381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:20.946600 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:20.946853 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:20.946859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:20.947891 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:20.947892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:20.949106 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:20.949108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:20.949652 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:20.949652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:20.949777 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:20.950316 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:20.950325 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:20.952097 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:20.953966 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:20.953972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:20.953975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:20.954229 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:20.954449 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:20.954449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:20.954450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:20.954573 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:20.954877 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:20.954882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:20.955054 (+ 172us) client_negotiation.cc:769] Sending connection context
0621 05:18:20.955709 (+ 655us) client_negotiation.cc:240] Negotiation successful
0621 05:18:20.955891 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":74,"threads_started":1}
I20260621 05:18:20.956735 12993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36909
I20260621 05:18:20.956818 12993 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:20.956983 12993 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:20.957370 12644 ts_manager.cc:194] Registered new tserver with Master: 1ab829b1cf684fa39da559ec93e14aca (127.0.98.2:37065)
I20260621 05:18:20.957835 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58883
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:18:20.995069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36909
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:21.109457 12999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:21.109760 12999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:21.109874 12999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:21.111600 12999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:21.111724 12999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:21.111810 12999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:18:21.113924 12999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:21.114387 12999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:21.114665 12999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:21.122970 13005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:21.122952 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:21.123386 13007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:21.123808 12999 server_base.cc:1034] running on GCE node
I20260621 05:18:21.123978 12999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:21.124197 12999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:21.125329 12999 hybrid_clock.cc:648] HybridClock initialized: now 1782019101125308 us; error 55 us; skew 500 ppm
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:21.128654 12999 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:18:21.129242 12999 webserver.cc:460] Webserver started at http://127.0.98.3:40463/ using document root <none> and password file <none>
I20260621 05:18:21.129561 12999 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:21.129683 12999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:21.129882 12999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:21.143067 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "7489f406599c48278456c23dfcb0b8a6"
format_stamp: "Formatted at 2026-06-21 05:18:21 on dist-test-slave-czm5"
server_key: "3cf93817d64a4e440e3d8fbddbb67d3d"
server_key_iv: "7860a12ce535890aacad7405eca593f7"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:21.143595 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7489f406599c48278456c23dfcb0b8a6"
format_stamp: "Formatted at 2026-06-21 05:18:21 on dist-test-slave-czm5"
server_key: "3cf93817d64a4e440e3d8fbddbb67d3d"
server_key_iv: "7860a12ce535890aacad7405eca593f7"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:21.148011 12999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260621 05:18:21.161123 13014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.161500 12999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:18:21.161616 12999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "7489f406599c48278456c23dfcb0b8a6"
format_stamp: "Formatted at 2026-06-21 05:18:21 on dist-test-slave-czm5"
server_key: "3cf93817d64a4e440e3d8fbddbb67d3d"
server_key_iv: "7860a12ce535890aacad7405eca593f7"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:21.161738 12999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:21.215389 12999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:21.217895 12999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:21.218088 12999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:21.218354 12999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:21.218751 12999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:21.218801 12999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.218830 12999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:21.218847 12999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.225839 12999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44383
I20260621 05:18:21.226279 12999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:18:21.230967 13127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44383 every 8 connection(s)
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:21.235654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 12999
I20260621 05:18:21.235750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:18:21.235991 392 external_mini_cluster.cc:1428] Setting key 16d3123dfc60646e2417a597f19c5717
I20260621 05:18:21.238914 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:21.227335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35657 (local address 127.0.98.62:36909)
0621 05:18:21.227389 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:21.227391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:21.229887 (+ 2496us) server_negotiation.cc:407] Connection header received
0621 05:18:21.230517 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:21.230518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:21.230530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:21.230643 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:21.231461 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.232046 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.233109 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.233393 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.233459 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:21.235606 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:21.235608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:21.235609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:21.235617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:21.237392 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:21.237883 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:21.237885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:21.237885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:21.237943 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:21.238189 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:21.238190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:21.238190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:21.238306 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:21.238386 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:21.238711 (+ 325us) server_negotiation.cc:299] Negotiation successful
0621 05:18:21.238809 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:18:21.239192 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:21.227069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36909 (local address 127.0.98.3:35657)
0621 05:18:21.229737 (+ 2668us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:21.229797 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:21.230404 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:21.230796 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:21.230801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:21.231076 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:21.231348 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:21.231353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.232159 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.232160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:21.233010 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:21.233012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.233479 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.233480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:21.233597 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:21.234135 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:21.234143 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:21.235492 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:21.237481 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:21.237487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:21.237489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:21.237791 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:21.238016 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:21.238017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:21.238017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:21.238116 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:21.238377 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:21.238383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:21.238539 (+ 156us) client_negotiation.cc:769] Sending connection context
0621 05:18:21.238645 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 05:18:21.238840 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2619,"thread_start_us":75,"threads_started":1}
I20260621 05:18:21.239733 13128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36909
I20260621 05:18:21.239856 13128 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:21.240031 13128 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:21.240454 12644 ts_manager.cc:194] Registered new tserver with Master: 7489f406599c48278456c23dfcb0b8a6 (127.0.98.3:44383)
I20260621 05:18:21.240850 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35657
I20260621 05:18:21.247699 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:18:21.248545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 12725
I20260621 05:18:21.254923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 12862
I20260621 05:18:21.261161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 12999
I20260621 05:18:21.267342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 12612
I20260621 05:18:21.272814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36909
--webserver_interface=127.0.98.62
--webserver_port=45433
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36909
--ranger_config_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33727/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33727/kms with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:21.350178 13131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:21.350373 13131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:21.350391 13131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:21.351634 13131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:21.351706 13131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:21.351730 13131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:21.351752 13131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:21.351781 13131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:21.353322 13131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33727/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36909
--ranger_config_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:21.353582 13131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:21.353791 13131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:21.358297 13136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:21.358320 13137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:21.358439 13131 server_base.cc:1034] running on GCE node
W20260621 05:18:21.358560 13139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:21.358763 13131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:21.359081 13131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:21.360241 13131 hybrid_clock.cc:648] HybridClock initialized: now 1782019101360224 us; error 30 us; skew 500 ppm
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:21.362627 13131 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:18:21.363065 13131 webserver.cc:460] Webserver started at http://127.0.98.62:45433/ using document root <none> and password file <none>
I20260621 05:18:21.363283 13131 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:21.363355 13131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:21.401947 13131 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:21.402585 13147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.402742 13131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:21.402834 13131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "bf0eaf386598445a90e11d209189a2de"
format_stamp: "Formatted at 2026-06-21 05:18:19 on dist-test-slave-czm5"
server_key: "075fac6dc0121ac409d47c0aba290710"
server_key_iv: "9a2ac50aa0978619087af38dafc40002"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:21.403185 13131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:21.413851 13131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:21.415167 13131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:21.415309 13131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:21.419332 13131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36909
I20260621 05:18:21.419409 13199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36909 every 8 connection(s)
I20260621 05:18:21.419773 13131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:18:21.420854 13200 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:18:21.421471 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Bootstrap starting.
I20260621 05:18:21.428251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 13131
I20260621 05:18:21.428504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43663
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36909
--webserver_port=42403
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33727/kms with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:21.429082 13200 log.cc:824] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:21.430984 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:18:21.431252 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Bootstrap complete.
I20260621 05:18:21.433552 13200 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } }
I20260621 05:18:21.433751 13200 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0eaf386598445a90e11d209189a2de, State: Initialized, Role: FOLLOWER
I20260621 05:18:21.433921 13200 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } }
I20260621 05:18:21.433992 13200 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:18:21.434037 13200 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:18:21.434085 13200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:18:21.434986 13200 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } }
I20260621 05:18:21.435067 13200 leader_election.cc:302] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0eaf386598445a90e11d209189a2de; no voters:
I20260621 05:18:21.435249 13200 leader_election.cc:288] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:18:21.435289 13204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:18:21.435434 13204 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [term 2 LEADER]: Becoming Leader. State: Replica: bf0eaf386598445a90e11d209189a2de, State: Running, Role: LEADER
I20260621 05:18:21.435493 13200 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:21.435583 13204 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } }
I20260621 05:18:21.435964 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0eaf386598445a90e11d209189a2de. Latest consensus state: current_term: 2 leader_uuid: "bf0eaf386598445a90e11d209189a2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } } }
I20260621 05:18:21.435971 13206 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bf0eaf386598445a90e11d209189a2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0eaf386598445a90e11d209189a2de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36909 } } }
I20260621 05:18:21.436226 13206 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:21.436136 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:21.436496 13213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:18:21.437074 13213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:18:21.437219 13213 catalog_manager.cc:1259] Loaded cluster ID: 7ae9a834866d406487b662952de7209e
I20260621 05:18:21.437273 13213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:18:21.437408 13200 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 05:18:21.440240 13213 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:18:21.440667 13213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Loaded TSK: 0
I20260621 05:18:21.440975 13213 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:18:21.537528 13203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:21.537823 13203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:21.537881 13203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:21.539471 13203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:21.539556 13203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:21.539613 13203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:18:21.541690 13203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42403
--tserver_master_addrs=127.0.98.62:36909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:21.541946 13203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:21.542241 13203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:21.548189 13240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:21.548218 13238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:21.548852 13237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:21.549153 13203 server_base.cc:1034] running on GCE node
I20260621 05:18:21.549424 13203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:21.549680 13203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:21.550868 13203 hybrid_clock.cc:648] HybridClock initialized: now 1782019101550837 us; error 45 us; skew 500 ppm
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:21.556052 13203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:18:21.556551 13203 webserver.cc:460] Webserver started at http://127.0.98.1:42403/ using document root <none> and password file <none>
I20260621 05:18:21.556792 13203 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:21.556856 13203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:21.577335 13203 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260621 05:18:21.579296 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.579506 13203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260621 05:18:21.579586 13203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "d0b6a4d4d79d4bbca51a3be431b0646c"
format_stamp: "Formatted at 2026-06-21 05:18:20 on dist-test-slave-czm5"
server_key: "6d49a0d9d497e86dd8c01a83dd170d24"
server_key_iv: "e4fdbeeccb1b6679483ec9f8de82037c"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:21.580025 13203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:21.617302 13203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:21.619681 13203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:21.619876 13203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:21.620151 13203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:21.620535 13203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:21.620620 13203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.620741 13203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:21.620821 13203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.627087 13203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43663
I20260621 05:18:21.627604 13203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:18:21.629746 13361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43663 every 8 connection(s)
I20260621 05:18:21.631289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 13203
I20260621 05:18:21.631580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37065
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36909
--webserver_port=34099
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33727/kms with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:21.645725 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:21.629927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36909 (local address 127.0.98.1:38969)
0621 05:18:21.633712 (+ 3785us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:21.633760 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:21.634398 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:21.635059 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:21.635063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:21.635296 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:21.635557 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:21.635563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.637245 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.637247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:21.638444 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:21.638447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.639329 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.639330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:21.639474 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:21.640097 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:21.640107 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:21.641812 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:21.643999 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:21.644005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:21.644007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:21.644270 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:21.644543 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:21.644544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:21.644544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:21.644661 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:21.645107 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:21.645112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:21.645268 (+ 156us) client_negotiation.cc:769] Sending connection context
0621 05:18:21.645361 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 05:18:21.645511 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3739,"thread_start_us":99,"threads_started":1}
I20260621 05:18:21.646575 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:21.630292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38969 (local address 127.0.98.62:36909)
0621 05:18:21.630503 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:21.630513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:21.634575 (+ 4062us) server_negotiation.cc:407] Connection header received
0621 05:18:21.634738 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:21.634744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:21.634797 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:21.634918 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:21.635718 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.637127 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.638553 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.639126 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.639225 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:21.641936 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:21.641941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:21.641941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:21.641952 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:21.643908 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:21.644359 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:21.644362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:21.644362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:21.644459 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:21.644731 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:21.644731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:21.644732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:21.645010 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:21.645081 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:21.646252 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0621 05:18:21.646369 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260621 05:18:21.647684 13362 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36909
I20260621 05:18:21.647773 13362 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:21.647936 13362 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
W20260621 05:18:21.648770 13362 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36909 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:18:21.648833 13362 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:21.648885 13362 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:21.649240 13362 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:21.649307 13362 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
W20260621 05:18:21.649607 13362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:18:21.744700 13366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:21.745076 13366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:21.745172 13366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:21.746815 13366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:21.746927 13366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:21.747015 13366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:18:21.749115 13366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34099
--tserver_master_addrs=127.0.98.62:36909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:21.749527 13366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:21.749794 13366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:21.756084 13372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:21.756481 13374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:21.756963 13371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:21.757668 13366 server_base.cc:1034] running on GCE node
I20260621 05:18:21.757861 13366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:21.758117 13366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:21.759266 13366 hybrid_clock.cc:648] HybridClock initialized: now 1782019101759259 us; error 29 us; skew 500 ppm
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:21.765240 13366 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:18:21.765743 13366 webserver.cc:460] Webserver started at http://127.0.98.2:34099/ using document root <none> and password file <none>
I20260621 05:18:21.766014 13366 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:21.766113 13366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:21.792016 13366 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.001s sys 0.000s
I20260621 05:18:21.792714 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.792903 13366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:21.793020 13366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "1ab829b1cf684fa39da559ec93e14aca"
format_stamp: "Formatted at 2026-06-21 05:18:20 on dist-test-slave-czm5"
server_key: "7e7bba600d3158aa71622e3ef645ef6e"
server_key_iv: "11f6a0abb4125e7435ecb43f40aeea45"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:21.793385 13366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:21.806840 13366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:21.809196 13366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:21.809387 13366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:21.809638 13366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:21.810004 13366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:21.810088 13366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.810159 13366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:21.810215 13366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.817564 13366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37065
I20260621 05:18:21.817801 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37065 every 8 connection(s)
I20260621 05:18:21.818073 13366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:21.827365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 13366
I20260621 05:18:21.827702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44383
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36909
--webserver_port=40463
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33727/kms with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:21.827888 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:21.819147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37719 (local address 127.0.98.62:36909)
0621 05:18:21.819205 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:21.819207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:21.819465 (+ 258us) server_negotiation.cc:407] Connection header received
0621 05:18:21.820046 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:21.820047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:21.820063 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:21.820115 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:21.820786 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.821297 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.822261 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.822528 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.822601 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:21.824809 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:21.824811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:21.824812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:21.824819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:21.826277 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:21.826735 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:21.826737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:21.826737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:21.826794 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:21.827042 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:21.827043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:21.827043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:21.827155 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:21.827245 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:21.827670 (+ 425us) server_negotiation.cc:299] Negotiation successful
0621 05:18:21.827756 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:18:21.828476 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:21.818838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36909 (local address 127.0.98.2:37719)
0621 05:18:21.819240 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:21.819385 (+ 145us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:21.819946 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:21.820185 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:21.820189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:21.820400 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:21.820659 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:21.820664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.821384 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.821385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:21.822164 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:21.822165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:21.822620 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:21.822621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:21.822750 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:21.823314 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:21.823324 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:21.824701 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:21.826397 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:21.826410 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:21.826412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:21.826635 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:21.826856 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:21.826857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:21.826858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:21.826971 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:21.827276 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:21.827283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:21.827463 (+ 180us) client_negotiation.cc:769] Sending connection context
0621 05:18:21.828022 (+ 559us) client_negotiation.cc:240] Negotiation successful
0621 05:18:21.828199 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":90,"threads_started":1}
I20260621 05:18:21.828871 13496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36909
I20260621 05:18:21.828971 13496 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:21.829169 13496 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
W20260621 05:18:21.829749 13496 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36909 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:18:21.829813 13496 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:21.829870 13496 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:21.830199 13496 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:21.830262 13496 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
W20260621 05:18:21.830515 13496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:18:21.933418 13499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:21.933799 13499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:21.933872 13499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:21.935496 13499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:21.935602 13499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:21.935724 13499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:18:21.938114 13499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40463
--tserver_master_addrs=127.0.98.62:36909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:21.938573 13499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:21.938858 13499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:21.945281 13506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:21.945571 13508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:21.946331 13499 server_base.cc:1034] running on GCE node
W20260621 05:18:21.947980 13505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:21.948266 13499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:21.948503 13499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:21.955869 13499 hybrid_clock.cc:648] HybridClock initialized: now 1782019101955839 us; error 54 us; skew 500 ppm
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:21.958914 13499 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:18:21.959367 13499 webserver.cc:460] Webserver started at http://127.0.98.3:40463/ using document root <none> and password file <none>
I20260621 05:18:21.959591 13499 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:21.959681 13499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:18:21 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:21.978978 13499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:21.979784 13516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.979961 13499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:21.980036 13499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "7489f406599c48278456c23dfcb0b8a6"
format_stamp: "Formatted at 2026-06-21 05:18:21 on dist-test-slave-czm5"
server_key: "3cf93817d64a4e440e3d8fbddbb67d3d"
server_key_iv: "7860a12ce535890aacad7405eca593f7"
server_key_version: "kuduclusterkey@0"
I20260621 05:18:21.980444 13499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:21.998680 13499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:22.000976 13499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:22.001183 13499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:22.001472 13499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:22.001879 13499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:22.001983 13499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:22.002058 13499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:22.002118 13499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:22.008488 13499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44383
I20260621 05:18:22.008932 13499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:18:22.010360 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44383 every 8 connection(s)
I20260621 05:18:22.011437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 13499
Jun 21 05:18:22 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:22.021605 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:22.011013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60387 (local address 127.0.98.62:36909)
0621 05:18:22.011057 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:22.011059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:22.011287 (+ 228us) server_negotiation.cc:407] Connection header received
0621 05:18:22.012241 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:22.012242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:22.012254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:22.012320 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:22.013050 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:22.013786 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:22.015252 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:22.015543 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:22.015586 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:22.018227 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:22.018228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:22.018228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:22.018235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:22.020157 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:22.020600 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:22.020601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:22.020601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:22.020655 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:22.020932 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:22.020933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:22.020933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:22.021052 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:22.021091 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:22.021437 (+ 346us) server_negotiation.cc:299] Negotiation successful
0621 05:18:22.021511 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:18:22.021981 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:22.011799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.0.98.62:36909)
0621 05:18:22.012478 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:22.012480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:22.012487 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:18:22.012508 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:22.012508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:22.012519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:22.012573 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:22.013246 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:22.013992 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:22.015098 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:22.015389 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:22.015465 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:22.017965 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:22.017967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:22.017967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:22.017974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:22.020389 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:22.020801 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:22.020802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:22.020803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:22.020866 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:22.021067 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:22.021068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:22.021068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:22.021189 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:22.021341 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:22.021811 (+ 470us) server_negotiation.cc:299] Negotiation successful
0621 05:18:22.021893 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":75,"threads_started":1}
I20260621 05:18:22.032392 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:22.011013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36909 (local address 127.0.98.3:60387)
0621 05:18:22.011160 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:22.011205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:22.012138 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:22.012450 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:22.012454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:22.012680 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:22.012942 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:22.012948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:22.013909 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:22.013910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:22.015153 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:22.015156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:22.015703 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:22.015703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:22.015873 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:22.016299 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:22.016309 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:22.018121 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:22.020244 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:22.020250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:22.020252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:22.020516 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:22.020732 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:22.020733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:22.020733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:22.020859 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:22.021133 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:22.021138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:22.021295 (+ 157us) client_negotiation.cc:769] Sending connection context
0621 05:18:22.023599 (+ 2304us) client_negotiation.cc:240] Negotiation successful
0621 05:18:22.024264 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260621 05:18:22.033078 13630 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36909
I20260621 05:18:22.033183 13630 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:22.033361 13630 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
W20260621 05:18:22.033975 13630 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36909 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:18:22.034042 13630 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:22.034101 13630 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:22.034613 13630 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:22.034689 13630 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
W20260621 05:18:22.035235 13630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 05:18:22 dist-test-slave-czm5 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:22.649785 13362 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:22.649927 13362 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:22.650305 13164 ts_manager.cc:194] Registered new tserver with Master: d0b6a4d4d79d4bbca51a3be431b0646c (127.0.98.1:43663)
I20260621 05:18:22.651319 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38969
I20260621 05:18:22.830675 13496 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:22.830833 13496 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:22.831101 13164 ts_manager.cc:194] Registered new tserver with Master: 1ab829b1cf684fa39da559ec93e14aca (127.0.98.2:37065)
I20260621 05:18:22.831663 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37719
Jun 21 05:18:22 dist-test-slave-czm5 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260621 05:18:23.035411 13630 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:23.035575 13630 heartbeater.cc:511] Master 127.0.98.62:36909 requested a full tablet report, sending...
I20260621 05:18:23.035955 13164 ts_manager.cc:194] Registered new tserver with Master: 7489f406599c48278456c23dfcb0b8a6 (127.0.98.3:44383)
I20260621 05:18:23.036437 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60387
I20260621 05:18:23.039819 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:18:23.048264 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.040740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.0.98.62:36909)
0621 05:18:23.041040 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:23.041042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:23.041192 (+ 150us) server_negotiation.cc:407] Connection header received
0621 05:18:23.041213 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:23.041213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:23.041224 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:23.041275 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:23.041941 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.042653 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.043576 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.043869 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.043920 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.045305 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:23.045306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:23.045307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:23.045313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:23.046908 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:23.047247 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:23.047248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:23.047248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:23.047302 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:23.047481 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:23.047481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:23.047482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:23.047589 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:23.047695 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:23.048099 (+ 404us) server_negotiation.cc:299] Negotiation successful
0621 05:18:23.048172 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":85,"threads_started":1}
I20260621 05:18:23.050540 13164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:18:23.052688 13164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:18:23.066823 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.057849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43663 (local address 127.0.0.1:46486)
0621 05:18:23.058400 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:23.058437 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:23.059104 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:23.060789 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:23.060790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:23.060792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:23.060843 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.060845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.062800 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.062800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.063790 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.063792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.066546 (+ 2754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.066547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.066621 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.066633 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 05:18:23.066694 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:18:23.066741 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":62,"threads_started":1}
I20260621 05:18:23.067512 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.058123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.0.98.2:37065)
0621 05:18:23.058350 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:23.058355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:23.058927 (+ 572us) server_negotiation.cc:407] Connection header received
0621 05:18:23.058961 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:23.058963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:23.058978 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:23.059034 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:23.059458 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.060711 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.064539 (+ 3828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.065235 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.065557 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.065648 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:23.067373 (+ 1725us) server_negotiation.cc:299] Negotiation successful
0621 05:18:23.067418 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":35,"threads_started":1}
I20260621 05:18:23.067309 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.058023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37065 (local address 127.0.0.1:60614)
0621 05:18:23.058803 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:23.058811 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:23.058860 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:23.059333 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:23.059336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:23.059337 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:23.059394 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.059396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.062007 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.062008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.064396 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.064398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.067109 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.067110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.067166 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.067170 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:18:23.067217 (+ 47us) client_negotiation.cc:240] Negotiation successful
0621 05:18:23.067250 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":51,"threads_started":1}
I20260621 05:18:23.068243 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.057944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.0.98.1:43663)
0621 05:18:23.058749 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:23.058753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:23.058763 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:18:23.059175 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:23.059178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:23.059191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:23.059261 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:23.060912 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.061905 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.065678 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.066370 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.066435 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.066513 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:23.068128 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0621 05:18:23.068176 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":45,"threads_started":1}
I20260621 05:18:23.068665 13430 tablet_service.cc:1467] Processing CreateTablet for tablet bd5918ba26c84ee69cb7ef7cc65b78c7 (DEFAULT_TABLE table=test-table [id=4ea547c2201b43a3be0bc5f62b19b691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:23.069036 13430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5918ba26c84ee69cb7ef7cc65b78c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:23.068953 13287 tablet_service.cc:1467] Processing CreateTablet for tablet bd5918ba26c84ee69cb7ef7cc65b78c7 (DEFAULT_TABLE table=test-table [id=4ea547c2201b43a3be0bc5f62b19b691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:23.069238 13287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5918ba26c84ee69cb7ef7cc65b78c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:23.072149 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.058157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44383 (local address 127.0.0.1:57794)
0621 05:18:23.067704 (+ 9547us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:23.067715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:23.067813 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:23.068879 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:23.068880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:23.068882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:23.068915 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.068916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.070060 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.070061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.071050 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.071052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.071954 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.071955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.072022 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.072027 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:18:23.072053 (+ 26us) client_negotiation.cc:240] Negotiation successful
0621 05:18:23.072087 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9529,"thread_start_us":48,"threads_started":1}
I20260621 05:18:23.072499 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.058241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.0.98.3:44383)
0621 05:18:23.068488 (+ 10247us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:23.068494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:23.068504 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:18:23.068537 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:23.068539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:23.068554 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:23.068982 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:23.069053 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.069962 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.071147 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.071832 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.072287 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.072369 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:23.072391 (+ 22us) server_negotiation.cc:299] Negotiation successful
0621 05:18:23.072425 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10204,"thread_start_us":51,"threads_started":1}
I20260621 05:18:23.073107 13564 tablet_service.cc:1467] Processing CreateTablet for tablet bd5918ba26c84ee69cb7ef7cc65b78c7 (DEFAULT_TABLE table=test-table [id=4ea547c2201b43a3be0bc5f62b19b691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:23.073396 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5918ba26c84ee69cb7ef7cc65b78c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:23.075778 13664 tablet_bootstrap.cc:492] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: Bootstrap starting.
I20260621 05:18:23.076505 13664 tablet_bootstrap.cc:654] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:23.077306 13664 log.cc:824] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:23.077495 13665 tablet_bootstrap.cc:492] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c: Bootstrap starting.
I20260621 05:18:23.078511 13665 tablet_bootstrap.cc:654] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:23.078831 13665 log.cc:824] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:23.079560 13664 tablet_bootstrap.cc:492] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: No bootstrap required, opened a new log
I20260621 05:18:23.079682 13664 ts_tablet_manager.cc:1397] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 05:18:23.082592 13664 raft_consensus.cc:348] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.082840 13664 raft_consensus.cc:374] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:23.082924 13664 raft_consensus.cc:729] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab829b1cf684fa39da559ec93e14aca, State: Initialized, Role: FOLLOWER
I20260621 05:18:23.083204 13664 consensus_queue.cc:260] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.083525 13664 ts_tablet_manager.cc:1428] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260621 05:18:23.083801 13666 tablet_bootstrap.cc:492] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: Bootstrap starting.
I20260621 05:18:23.084452 13666 tablet_bootstrap.cc:654] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:23.084625 13496 heartbeater.cc:503] Master 127.0.98.62:36909 was elected leader, sending a full tablet report...
I20260621 05:18:23.084790 13666 log.cc:824] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:23.085167 13665 tablet_bootstrap.cc:492] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c: No bootstrap required, opened a new log
I20260621 05:18:23.085220 13665 ts_tablet_manager.cc:1397] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260621 05:18:23.087589 13665 raft_consensus.cc:348] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.087824 13665 raft_consensus.cc:374] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:23.087908 13665 raft_consensus.cc:729] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b6a4d4d79d4bbca51a3be431b0646c, State: Initialized, Role: FOLLOWER
I20260621 05:18:23.088099 13665 consensus_queue.cc:260] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.088424 13665 ts_tablet_manager.cc:1428] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:18:23.088569 13362 heartbeater.cc:503] Master 127.0.98.62:36909 was elected leader, sending a full tablet report...
I20260621 05:18:23.090194 13666 tablet_bootstrap.cc:492] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: No bootstrap required, opened a new log
I20260621 05:18:23.090337 13666 ts_tablet_manager.cc:1397] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260621 05:18:23.092458 13666 raft_consensus.cc:348] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.092659 13666 raft_consensus.cc:374] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:23.092734 13666 raft_consensus.cc:729] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7489f406599c48278456c23dfcb0b8a6, State: Initialized, Role: FOLLOWER
I20260621 05:18:23.092895 13666 consensus_queue.cc:260] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.093168 13666 ts_tablet_manager.cc:1428] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:18:23.093312 13630 heartbeater.cc:503] Master 127.0.98.62:36909 was elected leader, sending a full tablet report...
I20260621 05:18:23.094733 13671 raft_consensus.cc:482] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:23.094861 13671 raft_consensus.cc:504] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.095980 13671 leader_election.cc:288] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ab829b1cf684fa39da559ec93e14aca (127.0.98.2:37065), 7489f406599c48278456c23dfcb0b8a6 (127.0.98.3:44383)
I20260621 05:18:23.103102 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.095929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35991 (local address 127.0.98.3:44383)
0621 05:18:23.095962 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:23.095964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:23.096290 (+ 326us) server_negotiation.cc:407] Connection header received
0621 05:18:23.096444 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:23.096445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:23.096456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:23.096487 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:23.097108 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.097812 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.101107 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.101670 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.101720 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.101731 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:23.103002 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0621 05:18:23.103043 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:18:23.103463 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.095737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44383 (local address 127.0.98.1:35991)
0621 05:18:23.096243 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:23.096251 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:23.096360 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:23.096980 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:23.096981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:23.096982 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:23.097038 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.097039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.097955 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.097956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.101009 (+ 3053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.101012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.101794 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.101795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.102928 (+ 1133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.102931 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:18:23.103324 (+ 393us) client_negotiation.cc:240] Negotiation successful
0621 05:18:23.103365 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":94,"threads_started":1}
I20260621 05:18:23.103806 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.095514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34669 (local address 127.0.98.2:37065)
0621 05:18:23.095550 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:23.095552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:23.096204 (+ 652us) server_negotiation.cc:407] Connection header received
0621 05:18:23.096608 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:23.096609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:23.096619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:23.096647 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:23.098036 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.098728 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.102067 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.102637 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.102682 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.102690 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:23.103712 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0621 05:18:23.103753 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:18:23.104146 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.095324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37065 (local address 127.0.98.1:34669)
0621 05:18:23.096141 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:23.096152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:23.096314 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:23.096791 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:23.096792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:23.096794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:23.096875 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.096877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.098827 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.098827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.099903 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:23.099906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.102782 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.102783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:23.102866 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.102870 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:18:23.104035 (+ 1165us) client_negotiation.cc:240] Negotiation successful
0621 05:18:23.104076 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":99,"threads_started":1}
I20260621 05:18:23.104416 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5918ba26c84ee69cb7ef7cc65b78c7" candidate_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7489f406599c48278456c23dfcb0b8a6" is_pre_election: true
I20260621 05:18:23.104547 13584 raft_consensus.cc:2454] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b6a4d4d79d4bbca51a3be431b0646c in term 0.
I20260621 05:18:23.104698 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5918ba26c84ee69cb7ef7cc65b78c7" candidate_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ab829b1cf684fa39da559ec93e14aca" is_pre_election: true
I20260621 05:18:23.104748 13250 leader_election.cc:302] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7489f406599c48278456c23dfcb0b8a6, d0b6a4d4d79d4bbca51a3be431b0646c; no voters:
I20260621 05:18:23.104806 13447 raft_consensus.cc:2454] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b6a4d4d79d4bbca51a3be431b0646c in term 0.
I20260621 05:18:23.104955 13671 raft_consensus.cc:2781] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:18:23.105042 13671 raft_consensus.cc:482] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:18:23.105098 13671 raft_consensus.cc:3037] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:23.105881 13671 raft_consensus.cc:504] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.106166 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5918ba26c84ee69cb7ef7cc65b78c7" candidate_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ab829b1cf684fa39da559ec93e14aca"
I20260621 05:18:23.106227 13447 raft_consensus.cc:3037] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:23.106947 13447 raft_consensus.cc:2454] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b6a4d4d79d4bbca51a3be431b0646c in term 1.
I20260621 05:18:23.107137 13249 leader_election.cc:302] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab829b1cf684fa39da559ec93e14aca, d0b6a4d4d79d4bbca51a3be431b0646c; no voters:
I20260621 05:18:23.107292 13672 raft_consensus.cc:2781] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:23.107345 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5918ba26c84ee69cb7ef7cc65b78c7" candidate_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7489f406599c48278456c23dfcb0b8a6"
I20260621 05:18:23.107410 13584 raft_consensus.cc:3037] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:23.107450 13672 raft_consensus.cc:686] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [term 1 LEADER]: Becoming Leader. State: Replica: d0b6a4d4d79d4bbca51a3be431b0646c, State: Running, Role: LEADER
I20260621 05:18:23.107527 13672 consensus_queue.cc:237] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } }
I20260621 05:18:23.108114 13584 raft_consensus.cc:2454] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b6a4d4d79d4bbca51a3be431b0646c in term 1.
I20260621 05:18:23.108220 13164 catalog_manager.cc:5495] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b6a4d4d79d4bbca51a3be431b0646c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b6a4d4d79d4bbca51a3be431b0646c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } health_report { overall_health: UNKNOWN } } }
I20260621 05:18:23.108341 13671 leader_election.cc:288] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [CANDIDATE]: Term 1 election: Requested vote from peers 1ab829b1cf684fa39da559ec93e14aca (127.0.98.2:37065), 7489f406599c48278456c23dfcb0b8a6 (127.0.98.3:44383)
I20260621 05:18:23.118854 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.115683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.0.98.1:43663)
0621 05:18:23.115719 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:23.115721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:23.115728 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:18:23.115847 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:23.115848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:23.115866 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:23.115950 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:18:23.116117 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.116765 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.117836 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:23.118120 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:23.118179 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:23.118372 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:18:23.118478 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:18:23.118545 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:23.118772 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 05:18:23.118792 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:18:23.120764 13584 raft_consensus.cc:1261] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 1 FOLLOWER]: Refusing update from remote peer d0b6a4d4d79d4bbca51a3be431b0646c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:23.121132 13676 consensus_queue.cc:1035] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7489f406599c48278456c23dfcb0b8a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:23.122998 13447 raft_consensus.cc:1261] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 1 FOLLOWER]: Refusing update from remote peer d0b6a4d4d79d4bbca51a3be431b0646c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:23.123296 13676 consensus_queue.cc:1035] T bd5918ba26c84ee69cb7ef7cc65b78c7 P d0b6a4d4d79d4bbca51a3be431b0646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab829b1cf684fa39da559ec93e14aca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:23.126623 13679 mvcc.cc:205] Tried to move back new op lower bound from 7299150246380253184 to 7299150246329184256. Current Snapshot: MvccSnapshot[applied={T|T < 7299150246380253184}]
I20260621 05:18:23.127149 13682 mvcc.cc:205] Tried to move back new op lower bound from 7299150246380253184 to 7299150246329184256. Current Snapshot: MvccSnapshot[applied={T|T < 7299150246380253184}]
I20260621 05:18:23.129127 13680 mvcc.cc:205] Tried to move back new op lower bound from 7299150246380253184 to 7299150246329184256. Current Snapshot: MvccSnapshot[applied={T|T < 7299150246380253184}]
I20260621 05:18:23.132208 13164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:18:23.132292 13164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:18:23.132964 13164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf0eaf386598445a90e11d209189a2de: Sending DeleteTablet for 3 replicas of tablet bd5918ba26c84ee69cb7ef7cc65b78c7
I20260621 05:18:23.133412 13287 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5918ba26c84ee69cb7ef7cc65b78c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46486
I20260621 05:18:23.133601 13430 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5918ba26c84ee69cb7ef7cc65b78c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60614
I20260621 05:18:23.133769 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5918ba26c84ee69cb7ef7cc65b78c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57794
I20260621 05:18:23.134341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 13203
I20260621 05:18:23.136996 13689 tablet_replica.cc:330] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: stopping tablet replica
I20260621 05:18:23.137121 13690 tablet_replica.cc:330] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: stopping tablet replica
I20260621 05:18:23.137276 13690 raft_consensus.cc:2229] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:18:23.137387 13689 raft_consensus.cc:2229] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:18:23.137526 13689 raft_consensus.cc:2258] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:23.137630 13690 raft_consensus.cc:2258] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:23.138168 13689 ts_tablet_manager.cc:1905] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:23.143298 13689 ts_tablet_manager.cc:1918] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:23.143410 13689 log.cc:1196] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/bd5918ba26c84ee69cb7ef7cc65b78c7
I20260621 05:18:23.143764 13689 ts_tablet_manager.cc:1939] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 1ab829b1cf684fa39da559ec93e14aca: Deleting consensus metadata
I20260621 05:18:23.144274 13690 ts_tablet_manager.cc:1905] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:23.145947 13690 ts_tablet_manager.cc:1918] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:23.146071 13690 log.cc:1196] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/bd5918ba26c84ee69cb7ef7cc65b78c7
I20260621 05:18:23.146384 13690 ts_tablet_manager.cc:1939] T bd5918ba26c84ee69cb7ef7cc65b78c7 P 7489f406599c48278456c23dfcb0b8a6: Deleting consensus metadata
I20260621 05:18:23.146812 13148 catalog_manager.cc:4841] TS 1ab829b1cf684fa39da559ec93e14aca (127.0.98.2:37065): tablet bd5918ba26c84ee69cb7ef7cc65b78c7 (table test-table [id=4ea547c2201b43a3be0bc5f62b19b691]) successfully deleted
I20260621 05:18:23.147322 13149 catalog_manager.cc:4841] TS 7489f406599c48278456c23dfcb0b8a6 (127.0.98.3:44383): tablet bd5918ba26c84ee69cb7ef7cc65b78c7 (table test-table [id=4ea547c2201b43a3be0bc5f62b19b691]) successfully deleted
W20260621 05:18:23.147751 13148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43663 (error 108)
I20260621 05:18:23.148403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 13366
I20260621 05:18:23.152087 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:23.151952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43663 (local address 127.0.0.1:46498)
0621 05:18:23.151997 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:23.152038 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260621 05:18:23.152388 13148 catalog_manager.cc:4584] TS d0b6a4d4d79d4bbca51a3be431b0646c (127.0.98.1:43663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd5918ba26c84ee69cb7ef7cc65b78c7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43663: connect: Connection refused (error 111)
I20260621 05:18:23.154388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 13499
I20260621 05:18:23.160816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 13131
I20260621 05:18:23.166430 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 05:18:23.511924 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 05:18:23.512045 392 mini_ranger.cc:66] Stopping Ranger...
I20260621 05:18:23.848839 392 mini_ranger.cc:68] Stopped Ranger
2026-06-21 05:18:23.849 UTC [12198] LOG: received smart shutdown request
2026-06-21 05:18:23.852 UTC [12198] LOG: background worker "logical replication launcher" (PID 12207) exited with exit code 1
2026-06-21 05:18:23.853 UTC [12202] LOG: shutting down
2026-06-21 05:18:23.994 UTC [12198] LOG: database system is shut down
2026-06-21T05:18:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45030 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:18:24 dist-test-slave-czm5 krb5kdc[13700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:18:24 dist-test-slave-czm5 krb5kdc[13700](info): set up 2 sockets
Jun 21 05:18:24 dist-test-slave-czm5 krb5kdc[13700](info): commencing operation
krb5kdc: starting...
W20260621 05:18:25.103369 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:18:25.211450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35539
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35539,127.0.98.61:45811,127.0.98.60:40961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:25.288146 13713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:25.288313 13713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:25.288333 13713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:25.289412 13713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:25.289456 13713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:25.289470 13713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:25.289482 13713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:25.289531 13713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:25.291060 13713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35539,127.0.98.61:45811,127.0.98.60:40961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:25.291306 13713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:25.291527 13713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:25.296128 13718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:25.296293 13721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:25.296363 13713 server_base.cc:1034] running on GCE node
W20260621 05:18:25.296293 13719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:25.296656 13713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:25.296875 13713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:25.298055 13713 hybrid_clock.cc:648] HybridClock initialized: now 1782019105298046 us; error 26 us; skew 500 ppm
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:25.300313 13713 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:18:25.300738 13713 webserver.cc:460] Webserver started at http://127.0.98.62:33485/ using document root <none> and password file <none>
I20260621 05:18:25.300940 13713 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:25.301008 13713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:25.301172 13713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:25.302125 13713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "08c593eb01984aacbf34f2713e3ea934"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "98a9756e11d131edff3b64d68761e951"
server_key_iv: "49d79ee3b5e06c722b9d40b980ecbc63"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.302474 13713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "08c593eb01984aacbf34f2713e3ea934"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "98a9756e11d131edff3b64d68761e951"
server_key_iv: "49d79ee3b5e06c722b9d40b980ecbc63"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.303732 13713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:18:25.304425 13728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:25.304590 13713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:25.304689 13713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "08c593eb01984aacbf34f2713e3ea934"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "98a9756e11d131edff3b64d68761e951"
server_key_iv: "49d79ee3b5e06c722b9d40b980ecbc63"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.304778 13713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:25.321389 13713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:25.323469 13713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:25.323668 13713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:25.327431 13713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35539
I20260621 05:18:25.327500 13780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35539 every 8 connection(s)
I20260621 05:18:25.327927 13713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:18:25.328613 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:25.329960 13781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:25.335340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 13713
I20260621 05:18:25.335434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:18:25.335598 392 external_mini_cluster.cc:1428] Setting key b2835f443bfb1bc7d5114efcad4bc37b
I20260621 05:18:25.338861 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.330626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.98.62:35539)
0621 05:18:25.330753 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.330767 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.330850 (+ 83us) server_negotiation.cc:407] Connection header received
0621 05:18:25.331324 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.331325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.331350 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.331426 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.331979 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.332776 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.333570 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.333864 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.333934 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.335911 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.335918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.335919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.335931 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.337392 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.337778 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.337781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.337781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.337853 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.338064 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.338065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.338065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.338244 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.338329 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.338513 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.338638 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260621 05:18:25.338863 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.330592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:60578)
0621 05:18:25.330736 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.330770 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.331209 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.331451 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.331453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.331752 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.331898 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.331903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.332852 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.332853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.333502 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.333504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.333932 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.333933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.333983 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.334336 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.334343 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.335783 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.337512 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.337518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.337519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.337693 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.337916 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.337917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.337917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.338002 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.338311 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.338316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.338370 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.338487 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.338695 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":90,"thread_start_us":79,"threads_started":1}
I20260621 05:18:25.339556 13781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:18:25.339888 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.339783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56338)
0621 05:18:25.339824 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.339860 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:18:25.340040 13732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
I20260621 05:18:25.340438 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.340352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56350)
0621 05:18:25.340396 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.340418 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:18:25.340539 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45811: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260621 05:18:25.356515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45811
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35539,127.0.98.61:45811,127.0.98.60:40961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:25.389818 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } attempt: 1
I20260621 05:18:25.390255 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.390126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56358)
0621 05:18:25.390177 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.390212 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 05:18:25.390635 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.390558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56368)
0621 05:18:25.390595 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.390618 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:18:25.390729 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45811: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
W20260621 05:18:25.431886 13788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:25.432152 13788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:25.432202 13788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:25.434134 13788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:25.434193 13788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:25.434221 13788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:25.434243 13788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:25.434269 13788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:25.436751 13788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35539,127.0.98.61:45811,127.0.98.60:40961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:25.436998 13788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:25.437296 13788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:25.442690 13794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:25.442674 13793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:25.442844 13788 server_base.cc:1034] running on GCE node
W20260621 05:18:25.442706 13796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:25.443136 13788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:25.443403 13788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:25.444558 13788 hybrid_clock.cc:648] HybridClock initialized: now 1782019105444538 us; error 31 us; skew 500 ppm
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:25.446910 13788 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260621 05:18:25.447286 13788 webserver.cc:460] Webserver started at http://127.0.98.61:37271/ using document root <none> and password file <none>
I20260621 05:18:25.447530 13788 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:25.447571 13788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:25.447708 13788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:25.448647 13788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/data/instance:
uuid: "a080c623756647c4ac1f63993d1dbf31"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "e88d7e029c9c0c139cf40463a4f4af02"
server_key_iv: "1fba4d2a09f3938f33033b0121aea1b1"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.448962 13788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/wal/instance:
uuid: "a080c623756647c4ac1f63993d1dbf31"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "e88d7e029c9c0c139cf40463a4f4af02"
server_key_iv: "1fba4d2a09f3938f33033b0121aea1b1"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.450145 13788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:18:25.450738 13803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:25.450891 13788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:25.450980 13788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/wal
uuid: "a080c623756647c4ac1f63993d1dbf31"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "e88d7e029c9c0c139cf40463a4f4af02"
server_key_iv: "1fba4d2a09f3938f33033b0121aea1b1"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.451076 13788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:25.458858 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } attempt: 2
I20260621 05:18:25.459226 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.459112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56370)
0621 05:18:25.459165 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.459199 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:18:25.459530 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.459454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56380)
0621 05:18:25.459491 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.459514 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:18:25.459666 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45811: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45811: connect: Connection refused (error 111)
I20260621 05:18:25.471777 13788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:25.473838 13788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:25.473953 13788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:25.477598 13788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45811
I20260621 05:18:25.477870 13855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45811 every 8 connection(s)
I20260621 05:18:25.477921 13788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
I20260621 05:18:25.478535 13856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:25.479982 13856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:18:25.480504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 13788
I20260621 05:18:25.480609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-1/wal/instance
I20260621 05:18:25.480789 392 external_mini_cluster.cc:1428] Setting key c2a75428b6b62639b6de2e498ede8528
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260621 05:18:25.489316 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.480601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.0.98.62:35539)
0621 05:18:25.480666 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.480668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.480876 (+ 208us) server_negotiation.cc:407] Connection header received
0621 05:18:25.481283 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.481284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.481299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.481364 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.482020 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.482495 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.483972 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.484216 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.484269 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.486521 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.486523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.486523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.486530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.487974 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.488368 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.488370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.488370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.488429 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.488680 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.488681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.488681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.488814 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.488877 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.489139 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.489231 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:18:25.489485 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.480587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:60582)
0621 05:18:25.480748 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.480786 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.481196 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.481447 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.481450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.481677 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.481922 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.481928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.482583 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.482584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.483895 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.483898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.484304 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.484305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.484450 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.484891 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.484901 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.486428 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.488052 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.488059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.488061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.488276 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.488502 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.488503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.488503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.488610 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.488870 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.488876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.489012 (+ 136us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.489118 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.489272 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260621 05:18:25.490026 13856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260621 05:18:25.497792 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.490248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56396)
0621 05:18:25.490295 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.490304 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.490339 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.490690 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.490690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.490792 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.490828 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.490829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.492008 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.492010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.492853 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.492855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.493308 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.493309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.493389 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.493717 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.493722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.495002 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.496774 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.496774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.496775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.496939 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.497159 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.497160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.497160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.497211 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.497479 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.497479 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.497514 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.497582 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.497670 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260621 05:18:25.497866 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.490362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.0.98.61:45811)
0621 05:18:25.490501 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.490505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.490518 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:18:25.490535 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.490537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.490548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.490634 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.490899 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.491920 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.492928 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.493218 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.493271 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.495087 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.495094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.495094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.495101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.496671 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.497041 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.497044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.497044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.497096 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.497287 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.497288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.497288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.497408 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.497498 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.497608 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.497731 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260621 05:18:25.498383 13856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:18:25.498737 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.498636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35300)
0621 05:18:25.498668 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.498719 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260621 05:18:25.498865 13804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
I20260621 05:18:25.499256 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.499169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35316)
0621 05:18:25.499213 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.499236 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:18:25.499361 13856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40961: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260621 05:18:25.505657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40961
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35539,127.0.98.61:45811,127.0.98.60:40961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:25.548586 13856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } attempt: 1
I20260621 05:18:25.549180 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.548981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35318)
0621 05:18:25.549090 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.549139 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 05:18:25.549562 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.549470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35334)
0621 05:18:25.549515 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.549542 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:18:25.549664 13856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40961: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
I20260621 05:18:25.550791 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } attempt: 3
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260621 05:18:25.557761 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.551128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.0.98.61:45811)
0621 05:18:25.551193 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.551195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.551206 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:18:25.551281 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.551282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.551293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.551342 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.551674 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.552194 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.552891 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.553100 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.553174 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.555279 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.555280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.555280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.555286 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.556725 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.557098 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.557099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.557099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.557158 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.557337 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.557338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.557338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.557431 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.557480 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.557629 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.557688 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:18:25.557804 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.551058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56400)
0621 05:18:25.551128 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.551138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.551214 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.551344 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.551345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.551507 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.551572 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.551574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.552288 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.552289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.552797 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.552799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.553172 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.553173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.553250 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.553702 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.553707 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.555202 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.556803 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.556804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.556804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.557016 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.557217 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.557218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.557219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.557280 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.557490 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.557491 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.557538 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.557605 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.557692 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260621 05:18:25.558239 13781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:18:25.558611 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.558497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35348)
0621 05:18:25.558532 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.558588 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260621 05:18:25.558962 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.558872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35350)
0621 05:18:25.558918 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.558944 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:18:25.559073 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40961: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
W20260621 05:18:25.584908 13863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:25.585146 13863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:25.585180 13863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:25.586266 13863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:25.586329 13863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:25.586355 13863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:25.586385 13863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:25.586408 13863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:25.587918 13863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35539,127.0.98.61:45811,127.0.98.60:40961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:25.588179 13863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:25.588387 13863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:18:25.590250 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } attempt: 1
I20260621 05:18:25.590655 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.590527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35360)
0621 05:18:25.590593 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.590630 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 05:18:25.590960 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.590860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35374)
0621 05:18:25.590902 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.590926 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260621 05:18:25.591044 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40961: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
W20260621 05:18:25.593209 13868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:25.593252 13863 server_base.cc:1034] running on GCE node
W20260621 05:18:25.593153 13869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:25.593153 13871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:25.593629 13863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:25.593884 13863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:25.595016 13863 hybrid_clock.cc:648] HybridClock initialized: now 1782019105595008 us; error 27 us; skew 500 ppm
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:25.597321 13863 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260621 05:18:25.597729 13863 webserver.cc:460] Webserver started at http://127.0.98.60:35893/ using document root <none> and password file <none>
I20260621 05:18:25.597926 13863 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:25.597987 13863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:25.598121 13863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:25.599045 13863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/data/instance:
uuid: "dcab988ce88d46c28ddc5fac7408d704"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "7681988c0b1a9a0236304f6bf572e01d"
server_key_iv: "44ea15f63708e2d0bc162c677aba55f9"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.599361 13863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/wal/instance:
uuid: "dcab988ce88d46c28ddc5fac7408d704"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "7681988c0b1a9a0236304f6bf572e01d"
server_key_iv: "44ea15f63708e2d0bc162c677aba55f9"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.600582 13863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:18:25.601241 13878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:25.601408 13863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:25.601504 13863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/wal
uuid: "dcab988ce88d46c28ddc5fac7408d704"
format_stamp: "Formatted at 2026-06-21 05:18:25 on dist-test-slave-czm5"
server_key: "7681988c0b1a9a0236304f6bf572e01d"
server_key_iv: "44ea15f63708e2d0bc162c677aba55f9"
server_key_version: "encryptionkey@0"
I20260621 05:18:25.601589 13863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:25.617906 13856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } attempt: 2
I20260621 05:18:25.618278 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.618169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35386)
0621 05:18:25.618222 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.618254 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 05:18:25.618578 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.618491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35394)
0621 05:18:25.618532 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.618554 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 05:18:25.618656 13856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40961: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40961: connect: Connection refused (error 111)
I20260621 05:18:25.626219 13863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:25.627893 13863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:25.628046 13863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:25.631922 13863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40961
I20260621 05:18:25.631965 13930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40961 every 8 connection(s)
I20260621 05:18:25.632290 13863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
I20260621 05:18:25.633051 13931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:25.634402 13931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:25.640292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 13863
I20260621 05:18:25.640394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018933478224-392-0/minicluster-data/master-2/wal/instance
I20260621 05:18:25.640578 392 external_mini_cluster.cc:1428] Setting key 5cabb2a62130b0281c1a6541df58ca37
I20260621 05:18:25.643008 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.635018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.98.62:35539)
0621 05:18:25.635076 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.635078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.635265 (+ 187us) server_negotiation.cc:407] Connection header received
0621 05:18:25.635739 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.635741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.635752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.635793 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.636422 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.636903 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.637878 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.638063 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.638147 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.640026 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.640033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.640033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.640039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.641622 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.642079 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.642079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.642080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.642125 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.642397 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.642397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.642398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.642517 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.642589 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.642847 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.642925 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:18:25.643162 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.635001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:60592)
0621 05:18:25.635154 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.635192 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.635609 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.635914 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.635922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.636141 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.636314 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.636318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.636995 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.636996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.637767 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.637769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.638155 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.638157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.638278 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.638667 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.638676 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.639921 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.641694 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.641700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.641702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.642002 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.642197 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.642198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.642199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.642324 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.642583 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.642589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.642714 (+ 125us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.642812 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.642950 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260621 05:18:25.643731 13931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260621 05:18:25.649394 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.640963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.0.98.60:40961)
0621 05:18:25.641129 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.641133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.641308 (+ 175us) server_negotiation.cc:407] Connection header received
0621 05:18:25.641362 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.641365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.641378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.641478 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.642170 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.642836 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.644006 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.644286 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.644334 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.646713 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.646720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.646720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.646726 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.648293 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.648626 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.648628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.648628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.648676 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.648866 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.648867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.648867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.649018 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.649174 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.649280 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.649334 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260621 05:18:25.650767 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.643953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45811 (local address 127.0.0.1:56416)
0621 05:18:25.643994 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.644003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.644038 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.644147 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.644147 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.644266 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.644304 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.644305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.645060 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.645061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.645787 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.645789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.646271 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.646272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.646337 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.646725 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.646729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.648111 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.649797 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.649798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.649799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.650021 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.650238 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.650239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.650239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.650308 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.650581 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.650582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.650610 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.650658 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.650717 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:18:25.650863 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.644005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.0.98.61:45811)
0621 05:18:25.644061 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.644063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.644071 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:18:25.644093 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.644094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.644102 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.644146 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.644403 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.644983 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.645859 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.646187 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.646265 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.648196 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.648197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.648198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.648210 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.649723 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.650114 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.650115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.650116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.650172 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.650375 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.650375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.650376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.650513 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.650586 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.650683 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.650748 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:18:25.651130 13931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:18:25.657640 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.650095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.0.98.62:35539)
0621 05:18:25.650127 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.650129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.650172 (+ 43us) server_negotiation.cc:407] Connection header received
0621 05:18:25.650235 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.650236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.650244 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.650304 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.650831 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.651478 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.652368 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.652525 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.652602 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.654870 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.654872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.654872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.654878 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.656566 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.656871 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.656871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.656872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.656928 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.657123 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.657123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.657124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.657204 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.657436 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.657467 (+ 31us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.657561 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:18:25.658133 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.651361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35412)
0621 05:18:25.651411 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.651421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.651468 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.651606 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.651606 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.651779 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.651802 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.651803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.652579 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.652580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.653293 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.653296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.653826 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.653827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.653876 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.654178 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.654184 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.655314 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.656947 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.656948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.656948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.657175 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.657376 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.657376 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.657377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.657462 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.657855 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.657856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.657915 (+ 59us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.657989 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.658080 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 05:18:25.658237 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.651392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.0.98.60:40961)
0621 05:18:25.651422 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.651423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.651456 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:18:25.651535 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.651536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.651545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.651594 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.651870 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.652511 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.653375 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.653742 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.653815 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.655399 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.655401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.655401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.655407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.656873 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.657249 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.657249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.657250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.657300 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.657550 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.657551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.657552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.657658 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.657738 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.658045 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.658150 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":15}
I20260621 05:18:25.659686 13931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704: Bootstrap starting.
I20260621 05:18:25.660593 13931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:25.660894 13931 log.cc:824] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:25.661679 13931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704: No bootstrap required, opened a new log
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260621 05:18:25.663357 13931 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.663498 13931 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:25.663559 13931 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcab988ce88d46c28ddc5fac7408d704, State: Initialized, Role: FOLLOWER
I20260621 05:18:25.663712 13931 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.663959 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.664017 13931 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:25.664050 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:18:25.665269 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.658286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.0.98.61:45811)
0621 05:18:25.658327 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.658329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.658336 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:18:25.658358 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.658358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.658366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.658421 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.658963 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.659369 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.660257 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.660407 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.660462 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.662684 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.662686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.662686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.662693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.664212 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.664528 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.664529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.664530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.664578 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.664801 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.664801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.664802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.664885 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.665063 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.665113 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.665184 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:18:25.666173 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } attempt: 2
W20260621 05:18:25.666352 13948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:18:25.666463 13948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:18:25.667119 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:18:25.666074 13863 master_runner.cc:186] Error getting master registration for 127.0.98.62:35539: OK, instance_id { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" instance_seqno: 1782019105326160 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:18:25.668820 13863 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:18:25.675343 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.666481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.98.60:40961)
0621 05:18:25.667705 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.667707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.667717 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:18:25.667740 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.667741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.667752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.667818 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.668148 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.668792 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.669811 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.670080 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.670150 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.672370 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.672371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.672372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.672378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.674170 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.674568 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.674569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.674569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.674625 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.674812 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.674813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.674813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.674959 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.675045 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.675195 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.675259 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1189}
I20260621 05:18:25.675345 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.666386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35422)
0621 05:18:25.666442 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.666453 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.666493 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.667845 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.667846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.668014 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.668056 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.668058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.668883 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.668884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.669723 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.669726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.670159 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.670160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.670210 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.670595 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.670600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.672263 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.674258 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.674259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.674260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.674484 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.674684 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.674685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.674685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.674743 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.675042 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.675043 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.675087 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.675171 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.675256 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:18:25.675468 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.668429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.0.98.62:35539)
0621 05:18:25.668473 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.668475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.668992 (+ 517us) server_negotiation.cc:407] Connection header received
0621 05:18:25.669033 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.669033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.669043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.669093 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.669779 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.670290 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.671252 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.671450 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.671520 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.672849 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.672850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.672850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.672857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.674287 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.674654 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.674655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.674655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.674712 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.674908 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.674909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.674909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.675029 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.675118 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.675298 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.675372 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:18:25.677680 13781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934: Bootstrap starting.
I20260621 05:18:25.678391 13781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:25.678678 13781 log.cc:824] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:25.679941 13781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934: No bootstrap required, opened a new log
I20260621 05:18:25.682250 13781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.682403 13781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:25.682443 13781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c593eb01984aacbf34f2713e3ea934, State: Initialized, Role: FOLLOWER
I20260621 05:18:25.682593 13781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.682888 13781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:25.683459 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.668698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.0.98.61:45811)
0621 05:18:25.668727 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.668729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.675745 (+ 7016us) server_negotiation.cc:407] Connection header received
0621 05:18:25.675886 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.675886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.675896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.675942 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.676616 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.677300 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.678264 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.678547 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.678586 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.680283 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.680285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.680285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.680291 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.682335 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.682712 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.682712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.682713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.682759 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.682937 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.682938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.682938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.683044 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.683105 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.683297 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.683365 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:18:25.684508 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.669169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.0.98.60:40961)
0621 05:18:25.675717 (+ 6548us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.675719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.676497 (+ 778us) server_negotiation.cc:407] Connection header received
0621 05:18:25.676596 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.676597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.676607 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.676650 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.677344 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.678500 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.679304 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.679611 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.679692 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.680993 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.680994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.680995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.681001 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.682765 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.683194 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.683195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.683195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.683247 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.683456 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.683457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.683457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.684040 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.684132 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.684317 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.684394 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6529,"thread_start_us":85,"threads_started":1}
W20260621 05:18:25.685465 13970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:18:25.685503 13970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:18:25.685670 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.685729 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:18:25.685405 13713 master_runner.cc:186] Error getting master registration for 127.0.98.61:45811: OK, instance_id { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" instance_seqno: 1782019105476633 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:18:25.687335 13713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:18:25.689131 13937 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:25.689213 13937 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.689438 13937 leader_election.cc:288] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08c593eb01984aacbf34f2713e3ea934 (127.0.98.62:35539), a080c623756647c4ac1f63993d1dbf31 (127.0.98.61:45811)
I20260621 05:18:25.689977 13755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcab988ce88d46c28ddc5fac7408d704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c593eb01984aacbf34f2713e3ea934" is_pre_election: true
I20260621 05:18:25.690092 13755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcab988ce88d46c28ddc5fac7408d704 in term 0.
I20260621 05:18:25.690402 13882 leader_election.cc:302] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c593eb01984aacbf34f2713e3ea934, dcab988ce88d46c28ddc5fac7408d704; no voters:
I20260621 05:18:25.690548 13937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:18:25.690624 13937 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:18:25.690654 13937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:25.691526 13937 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.691663 13937 leader_election.cc:288] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [CANDIDATE]: Term 1 election: Requested vote from peers 08c593eb01984aacbf34f2713e3ea934 (127.0.98.62:35539), a080c623756647c4ac1f63993d1dbf31 (127.0.98.61:45811)
I20260621 05:18:25.691843 13755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcab988ce88d46c28ddc5fac7408d704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c593eb01984aacbf34f2713e3ea934"
I20260621 05:18:25.691913 13755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:25.692739 13755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcab988ce88d46c28ddc5fac7408d704 in term 1.
I20260621 05:18:25.692891 13882 leader_election.cc:302] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c593eb01984aacbf34f2713e3ea934, dcab988ce88d46c28ddc5fac7408d704; no voters:
I20260621 05:18:25.692971 13937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:25.693100 13937 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [term 1 LEADER]: Becoming Leader. State: Replica: dcab988ce88d46c28ddc5fac7408d704, State: Running, Role: LEADER
I20260621 05:18:25.693192 13937 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.693589 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab988ce88d46c28ddc5fac7408d704. Latest consensus state: current_term: 1 leader_uuid: "dcab988ce88d46c28ddc5fac7408d704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.693825 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:25.694097 13974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:18:25.694270 13974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:18:25.695689 13755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [term 1 FOLLOWER]: Refusing update from remote peer dcab988ce88d46c28ddc5fac7408d704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:25.696362 13972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:25.697561 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab988ce88d46c28ddc5fac7408d704. Latest consensus state: current_term: 1 leader_uuid: "dcab988ce88d46c28ddc5fac7408d704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.697645 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:18:25.698549 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcab988ce88d46c28ddc5fac7408d704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.698624 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:25.698828 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcab988ce88d46c28ddc5fac7408d704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.698897 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:18:25.699331 13976 mvcc.cc:205] Tried to move back new op lower bound from 7299150256927252480 to 7299150256919932928. Current Snapshot: MvccSnapshot[applied={T|T < 7299150256927252480 or (T in {7299150256927252480})}]
I20260621 05:18:25.699831 13974 catalog_manager.cc:1347] Generated new cluster ID: 57373e3803fe4258b16ac1cb017616d4
I20260621 05:18:25.699874 13974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:18:25.709790 13856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } attempt: 3
Jun 21 05:18:25 dist-test-slave-czm5 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:18:25.714391 13974 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:18:25.715220 13974 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:18:25.717514 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.710035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40961 (local address 127.0.0.1:35444)
0621 05:18:25.710119 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:25.710130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:25.710171 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:25.710339 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:25.710340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:25.710505 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:25.710547 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.710548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.711471 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.711472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.712292 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:25.712294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.712804 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.712804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:25.712857 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.713293 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:25.713298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:25.714712 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:25.716524 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.716524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.716525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.716723 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.716909 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:25.716909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:25.716910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:25.716975 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:25.717244 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:25.717244 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:25.717282 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 05:18:25.717340 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:18:25.717423 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 05:18:25.717870 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:25.710114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.0.98.60:40961)
0621 05:18:25.710154 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:25.710156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:25.710166 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:18:25.710239 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:25.710240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:25.710250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:25.710303 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:25.710619 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.711378 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.712385 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:25.712696 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:25.712754 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:25.714813 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:25.714814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:25.714815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:25.714822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:25.716432 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.716795 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.716796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.716797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.716845 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:25.717052 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:25.717053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:25.717053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:25.717166 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:25.717219 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:25.717680 (+ 461us) server_negotiation.cc:299] Negotiation successful
0621 05:18:25.717788 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:18:25.719306 13856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31: Bootstrap starting.
I20260621 05:18:25.720700 13856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:25.720980 13856 log.cc:824] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:25.721901 13856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31: No bootstrap required, opened a new log
I20260621 05:18:25.723909 13856 raft_consensus.cc:348] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.724035 13856 raft_consensus.cc:374] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:25.724073 13856 raft_consensus.cc:729] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a080c623756647c4ac1f63993d1dbf31, State: Initialized, Role: FOLLOWER
I20260621 05:18:25.724190 13856 consensus_queue.cc:260] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } }
I20260621 05:18:25.724447 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.724593 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:18:25.724807 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcab988ce88d46c28ddc5fac7408d704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a080c623756647c4ac1f63993d1dbf31" is_pre_election: true
I20260621 05:18:25.724822 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcab988ce88d46c28ddc5fac7408d704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a080c623756647c4ac1f63993d1dbf31"
I20260621 05:18:25.724822 13828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:25.725633 13974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704: Generated new TSK 0
I20260621 05:18:25.725766 13974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:18:25.726265 13828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [term 1 FOLLOWER]: Refusing update from remote peer dcab988ce88d46c28ddc5fac7408d704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:25.726548 13972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:25.724720 13856 sys_catalog.cc:564] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:25.728278 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab988ce88d46c28ddc5fac7408d704. Latest consensus state: current_term: 1 leader_uuid: "dcab988ce88d46c28ddc5fac7408d704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.728380 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:18:25.728607 13984 sys_catalog.cc:455] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcab988ce88d46c28ddc5fac7408d704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:25.728778 13984 sys_catalog.cc:458] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:18:25.729862 13998 catalog_manager.cc:1259] Loaded cluster ID: 57373e3803fe4258b16ac1cb017616d4
I20260621 05:18:25.729950 13998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31: loading cluster ID for follower catalog manager: success
I20260621 05:18:25.731097 13998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31: acquiring CA information for follower catalog manager: success
I20260621 05:18:25.731441 13998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a080c623756647c4ac1f63993d1dbf31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:18:26.316640 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcab988ce88d46c28ddc5fac7408d704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c593eb01984aacbf34f2713e3ea934" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "a080c623756647c4ac1f63993d1dbf31" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45811 } } peers { permanent_uuid: "dcab988ce88d46c28ddc5fac7408d704" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40961 } } }
I20260621 05:18:26.316788 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab988ce88d46c28ddc5fac7408d704 [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:26.685942 13970 catalog_manager.cc:1259] Loaded cluster ID: 57373e3803fe4258b16ac1cb017616d4
I20260621 05:18:26.686043 13970 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934: loading cluster ID for follower catalog manager: success
I20260621 05:18:26.686859 13970 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934: acquiring CA information for follower catalog manager: success
I20260621 05:18:26.687161 13970 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08c593eb01984aacbf34f2713e3ea934: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:18:26.783797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 13713
I20260621 05:18:26.789022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 13788
I20260621 05:18:26.793541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 13863
2026-06-21T05:18:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2692 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:18:26 dist-test-slave-czm5 krb5kdc[14009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:18:26 dist-test-slave-czm5 krb5kdc[14009](info): set up 2 sockets
Jun 21 05:18:26 dist-test-slave-czm5 krb5kdc[14009](info): commencing operation
krb5kdc: starting...
W20260621 05:18:27.807459 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:18:27 dist-test-slave-czm5 krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:18:27.911803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:27.986927 14022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:27.987107 14022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:27.987126 14022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:27.988291 14022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:27.988329 14022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:27.988343 14022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:27.988354 14022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:27.988399 14022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:27.989765 14022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:27.989995 14022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:27.990218 14022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:27.999287 14027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:27.999303 14030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:27.999344 14028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:28.000115 14022 server_base.cc:1034] running on GCE node
I20260621 05:18:28.000346 14022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:28.000604 14022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:28.001765 14022 hybrid_clock.cc:648] HybridClock initialized: now 1782019108001754 us; error 37 us; skew 500 ppm
Jun 21 05:18:28 dist-test-slave-czm5 krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:28.004205 14022 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:18:28.004606 14022 webserver.cc:460] Webserver started at http://127.0.98.62:36351/ using document root <none> and password file <none>
I20260621 05:18:28.004827 14022 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:28.004902 14022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:28.005048 14022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:28.006052 14022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "61d0638060cb47cb93c6d10b84482689"
format_stamp: "Formatted at 2026-06-21 05:18:28 on dist-test-slave-czm5"
server_key: "3b65d5e66191c8c39151cb011d867248"
server_key_iv: "0d171242fef76105649e710162a15ef6"
server_key_version: "encryptionkey@0"
I20260621 05:18:28.006390 14022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "61d0638060cb47cb93c6d10b84482689"
format_stamp: "Formatted at 2026-06-21 05:18:28 on dist-test-slave-czm5"
server_key: "3b65d5e66191c8c39151cb011d867248"
server_key_iv: "0d171242fef76105649e710162a15ef6"
server_key_version: "encryptionkey@0"
I20260621 05:18:28.007570 14022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:18:28.008292 14037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:28.008438 14022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:18:28.008533 14022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "61d0638060cb47cb93c6d10b84482689"
format_stamp: "Formatted at 2026-06-21 05:18:28 on dist-test-slave-czm5"
server_key: "3b65d5e66191c8c39151cb011d867248"
server_key_iv: "0d171242fef76105649e710162a15ef6"
server_key_version: "encryptionkey@0"
I20260621 05:18:28.008610 14022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:28.017563 14022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:28.019177 14022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:28.019328 14022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:28.023115 14022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41019
I20260621 05:18:28.023147 14089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41019 every 8 connection(s)
I20260621 05:18:28.023450 14022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:18:28.024293 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:28.025739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 14022
I20260621 05:18:28.025872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:18:28.026029 392 external_mini_cluster.cc:1428] Setting key 114fffcc4bbbe2e9bb7be12b37ac5862
I20260621 05:18:28.026600 14090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689: Bootstrap starting.
I20260621 05:18:28.027133 14090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:28.027406 14090 log.cc:824] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:28.028100 14090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689: No bootstrap required, opened a new log
I20260621 05:18:28.029742 14090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d0638060cb47cb93c6d10b84482689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } }
I20260621 05:18:28.029883 14090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:28.029927 14090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d0638060cb47cb93c6d10b84482689, State: Initialized, Role: FOLLOWER
I20260621 05:18:28.030051 14090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d0638060cb47cb93c6d10b84482689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } }
I20260621 05:18:28.030141 14090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:18:28.030180 14090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:18:28.030222 14090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:28.030697 14090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d0638060cb47cb93c6d10b84482689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } }
I20260621 05:18:28.030814 14090 leader_election.cc:302] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61d0638060cb47cb93c6d10b84482689; no voters:
I20260621 05:18:28.031006 14090 leader_election.cc:288] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:18:28.031064 14095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:28.031211 14095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [term 1 LEADER]: Becoming Leader. State: Replica: 61d0638060cb47cb93c6d10b84482689, State: Running, Role: LEADER
I20260621 05:18:28.031251 14090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:28.031303 14095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d0638060cb47cb93c6d10b84482689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } }
I20260621 05:18:28.031595 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61d0638060cb47cb93c6d10b84482689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d0638060cb47cb93c6d10b84482689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } } }
I20260621 05:18:28.031728 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:28.031585 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61d0638060cb47cb93c6d10b84482689. Latest consensus state: current_term: 1 leader_uuid: "61d0638060cb47cb93c6d10b84482689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d0638060cb47cb93c6d10b84482689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } } }
I20260621 05:18:28.031836 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689 [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:28.032032 14102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:18:28.032552 14102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:18:28 dist-test-slave-czm5 krb5kdc[14009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:28.034058 14102 catalog_manager.cc:1347] Generated new cluster ID: e8d22b7dae9e40108d5a229e60404b6b
I20260621 05:18:28.034335 14102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:18:28.036422 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:28.026495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.0.98.62:41019)
0621 05:18:28.026656 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:28.026664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:28.026696 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:18:28.027101 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:28.027107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:28.027148 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:28.027297 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:28.028065 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:28.028944 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:28.029713 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:28.030271 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:28.030382 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:28.033277 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:28.033281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:28.033281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:28.033290 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:28.034923 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:28.035263 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:28.035265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:28.035266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:28.035350 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:28.035553 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:28.035553 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:28.035554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:28.035819 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:28.036004 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:28.036079 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:18:28.036221 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260621 05:18:28.037242 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:18:28.058164 14102 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:18:28.058815 14102 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:18:28.063546 14102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61d0638060cb47cb93c6d10b84482689: Generated new TSK 0
I20260621 05:18:28.063717 14102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:18:28.090574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 14022
2026-06-21T05:18:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1295 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:18:28 dist-test-slave-czm5 krb5kdc[14117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:18:28 dist-test-slave-czm5 krb5kdc[14117](info): set up 2 sockets
Jun 21 05:18:28 dist-test-slave-czm5 krb5kdc[14117](info): commencing operation
krb5kdc: starting...
W20260621 05:18:29.087571 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:18:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:18:29.192620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39703
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:29.267405 14130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:29.267575 14130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:29.267593 14130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:29.268623 14130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:29.268659 14130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:29.268672 14130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:29.268683 14130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:29.268693 14130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:29.269950 14130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:29.270139 14130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:29.270292 14130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:29.274758 14138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:29.274721 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:29.274741 14136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:29.274993 14130 server_base.cc:1034] running on GCE node
I20260621 05:18:29.275228 14130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:29.275506 14130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:29.276662 14130 hybrid_clock.cc:648] HybridClock initialized: now 1782019109276653 us; error 42 us; skew 500 ppm
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:29.278879 14130 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:18:29.279278 14130 webserver.cc:460] Webserver started at http://127.0.98.62:39705/ using document root <none> and password file <none>
I20260621 05:18:29.279500 14130 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:29.279562 14130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:29.279714 14130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:29.280640 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "a2bdc27fffeb4ea1a49e5eb73681356e"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "3fef888d309d1d76b2c7f782bf407f81"
server_key_iv: "7291ab587cbc114ec03e04d71f21c3b3"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.280989 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2bdc27fffeb4ea1a49e5eb73681356e"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "3fef888d309d1d76b2c7f782bf407f81"
server_key_iv: "7291ab587cbc114ec03e04d71f21c3b3"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.282279 14130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:18:29.282990 14145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.283178 14130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:29.283269 14130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "a2bdc27fffeb4ea1a49e5eb73681356e"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "3fef888d309d1d76b2c7f782bf407f81"
server_key_iv: "7291ab587cbc114ec03e04d71f21c3b3"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.283352 14130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:29.300729 14130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:29.302953 14130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:29.303118 14130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:29.306922 14130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39703
I20260621 05:18:29.306962 14197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39703 every 8 connection(s)
I20260621 05:18:29.307301 14130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:18:29.308267 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:29.310364 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Bootstrap starting.
I20260621 05:18:29.310889 14198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:29.311161 14198 log.cc:824] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:29.311870 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: No bootstrap required, opened a new log
I20260621 05:18:29.313390 14198 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:29.313513 14198 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:29.313567 14198 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bdc27fffeb4ea1a49e5eb73681356e, State: Initialized, Role: FOLLOWER
I20260621 05:18:29.313699 14198 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:29.313755 14198 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:18:29.313774 14198 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:18:29.313839 14198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:29.314594 14198 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:29.314702 14198 leader_election.cc:302] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2bdc27fffeb4ea1a49e5eb73681356e; no voters:
I20260621 05:18:29.314878 14198 leader_election.cc:288] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:18:29.314914 14201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:29.315062 14201 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 1 LEADER]: Becoming Leader. State: Replica: a2bdc27fffeb4ea1a49e5eb73681356e, State: Running, Role: LEADER
I20260621 05:18:29.315127 14198 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:29.315147 14201 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:29.315523 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2bdc27fffeb4ea1a49e5eb73681356e. Latest consensus state: current_term: 1 leader_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } } }
I20260621 05:18:29.315533 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } } }
I20260621 05:18:29.315723 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:29.315707 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:29.316166 14208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:18:29.316673 14208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:18:29.317137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 14130
I20260621 05:18:29.317242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:18:29.317422 392 external_mini_cluster.cc:1428] Setting key 15c5a2a71ab7375c98eddda8956a55ab
I20260621 05:18:29.318404 14208 catalog_manager.cc:1347] Generated new cluster ID: c33d6ec7334042d29abef11b347ccffb
I20260621 05:18:29.318455 14208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:29.326709 14208 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:18:29.327390 14208 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:18:29.329887 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.317954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.0.98.62:39703)
0621 05:18:29.318120 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.318128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.318159 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:18:29.318259 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.318265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.318317 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.319813 (+ 1496us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:29.320659 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.322038 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.323145 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.323732 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.323824 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.326492 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:29.326495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:29.326496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:29.326506 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:29.328371 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.328758 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.328761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.328761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.328850 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.329039 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.329040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.329040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.329251 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:29.329391 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.329538 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.329664 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260621 05:18:29.334928 14208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Generated new TSK 0
I20260621 05:18:29.335085 14208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:18:29.363672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:29.450251 14224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:29.450433 14224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:29.450456 14224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:29.451508 14224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:29.451541 14224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:29.451567 14224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:18:29.452977 14224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:29.453148 14224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:29.453310 14224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:29.457877 14229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:29.457886 14230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:29.457886 14232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:29.458245 14224 server_base.cc:1034] running on GCE node
I20260621 05:18:29.458416 14224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:29.458600 14224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:29.459676 14224 hybrid_clock.cc:648] HybridClock initialized: now 1782019109459667 us; error 29 us; skew 500 ppm
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:29.462016 14224 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:18:29.462409 14224 webserver.cc:460] Webserver started at http://127.0.98.1:33233/ using document root <none> and password file <none>
I20260621 05:18:29.462646 14224 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:29.462736 14224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:29.462863 14224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:29.464133 14224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "71198d3948694a8a97bed7e98ac29189"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "bde24245807fd77a3e49cceb1b9d657b"
server_key_iv: "4453ccc6117bd0a3b62e2e87397c120b"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.464524 14224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71198d3948694a8a97bed7e98ac29189"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "bde24245807fd77a3e49cceb1b9d657b"
server_key_iv: "4453ccc6117bd0a3b62e2e87397c120b"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.465687 14224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:18:29.466355 14239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.466516 14224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:29.466600 14224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "71198d3948694a8a97bed7e98ac29189"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "bde24245807fd77a3e49cceb1b9d657b"
server_key_iv: "4453ccc6117bd0a3b62e2e87397c120b"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.466681 14224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:29.475819 14224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:29.477929 14224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:29.478080 14224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:29.478276 14224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:29.478535 14224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:29.478586 14224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.478623 14224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:29.478649 14224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.484668 14224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35049
I20260621 05:18:29.484740 14352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35049 every 8 connection(s)
I20260621 05:18:29.485000 14224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:18:29.487859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 14224
I20260621 05:18:29.487962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:18:29.488139 392 external_mini_cluster.cc:1428] Setting key 97c8686faa55fd501463e6c131b74f51
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:29.494804 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.485693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:48477)
0621 05:18:29.485850 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.485887 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:29.486282 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:29.486488 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:29.486491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:29.487053 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:29.487227 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.487232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.487866 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.487867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.488989 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.488992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.489407 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.489409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.489533 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.490014 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:29.490024 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:29.491603 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:29.493591 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:29.493597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:29.493598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:29.493800 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:29.494014 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:29.494015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:29.494015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:29.494101 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:29.494374 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:29.494378 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:29.494477 (+ 99us) client_negotiation.cc:769] Sending connection context
0621 05:18:29.494555 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:18:29.494663 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260621 05:18:29.494809 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.485739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48477 (local address 127.0.98.62:39703)
0621 05:18:29.485791 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.485792 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.485963 (+ 171us) server_negotiation.cc:407] Connection header received
0621 05:18:29.486358 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.486359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.486368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.486413 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:29.487296 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.487774 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.489095 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.489328 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.489400 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.491751 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:29.491752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:29.491752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:29.491760 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:29.493498 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.493872 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.493873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.493874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.493938 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.494165 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.494166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.494166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.494308 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:29.494380 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.494602 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.494700 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:18:29.495311 14353 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260621 05:18:29.495407 14353 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:29.495590 14353 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
I20260621 05:18:29.496116 14162 ts_manager.cc:194] Registered new tserver with Master: 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260621 05:18:29.496767 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:18:29.512377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:29.588397 14358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:29.588583 14358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:29.588605 14358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:29.589658 14358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:29.589699 14358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:29.589748 14358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:18:29.591097 14358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:29.591311 14358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:29.591527 14358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:29.596107 14366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:29.596105 14364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:29.596190 14363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:29.596212 14358 server_base.cc:1034] running on GCE node
I20260621 05:18:29.596444 14358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:29.596678 14358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:29.597811 14358 hybrid_clock.cc:648] HybridClock initialized: now 1782019109597801 us; error 41 us; skew 500 ppm
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:29.600023 14358 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:18:29.600358 14358 webserver.cc:460] Webserver started at http://127.0.98.2:36183/ using document root <none> and password file <none>
I20260621 05:18:29.600507 14358 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:29.600543 14358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:29.600618 14358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:29.601469 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "c241b29957d04ce0bca52ca5acb68487"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "7a270fe2e5dbc3bff83e028840c76203"
server_key_iv: "ddc07c8d18e87702eaebc3825a4f51c0"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.601763 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c241b29957d04ce0bca52ca5acb68487"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "7a270fe2e5dbc3bff83e028840c76203"
server_key_iv: "ddc07c8d18e87702eaebc3825a4f51c0"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.602949 14358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:18:29.603533 14373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.603710 14358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.603772 14358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "c241b29957d04ce0bca52ca5acb68487"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "7a270fe2e5dbc3bff83e028840c76203"
server_key_iv: "ddc07c8d18e87702eaebc3825a4f51c0"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.603819 14358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:29.611531 14358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:29.613188 14358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:29.613284 14358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:29.613443 14358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:29.613720 14358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:29.613751 14358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.613770 14358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:29.613816 14358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.619491 14358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34943
I20260621 05:18:29.619544 14486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34943 every 8 connection(s)
I20260621 05:18:29.619817 14358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:29.626513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 14358
I20260621 05:18:29.626621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:18:29.626804 392 external_mini_cluster.cc:1428] Setting key 500d25c8cff1e995d21428a26aed4829
I20260621 05:18:29.628410 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.620463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55147 (local address 127.0.98.62:39703)
0621 05:18:29.620551 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.620553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.620693 (+ 140us) server_negotiation.cc:407] Connection header received
0621 05:18:29.621120 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.621121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.621131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.621179 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:29.622007 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.622479 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.623373 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.623540 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.623586 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.625494 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:29.625495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:29.625496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:29.625502 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:29.626951 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.627416 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.627418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.627418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.627480 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.627791 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.627792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.627792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.627907 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:29.627982 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.628243 (+ 261us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.628322 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:18:29.628608 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.620423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:55147)
0621 05:18:29.620588 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.620619 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:29.621026 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:29.621228 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:29.621231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:29.621764 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:29.621933 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.621937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.622555 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.622555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.623305 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.623306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.623606 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.623607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.623771 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.624167 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:29.624175 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:29.625413 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:29.627095 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:29.627102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:29.627104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:29.627321 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:29.627545 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:29.627546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:29.627547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:29.627721 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:29.627974 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:29.627980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:29.628115 (+ 135us) client_negotiation.cc:769] Sending connection context
0621 05:18:29.628222 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:18:29.628368 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260621 05:18:29.629112 14487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260621 05:18:29.629212 14487 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:29.629374 14487 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
I20260621 05:18:29.629745 14162 ts_manager.cc:194] Registered new tserver with Master: c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943)
I20260621 05:18:29.630267 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55147
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:18:29.648550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:29.724162 14492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:29.724339 14492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:29.724359 14492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:29.725370 14492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:29.725399 14492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:29.725421 14492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:18:29.726671 14492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:29.726842 14492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:29.726995 14492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:29.731614 14498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:29.731627 14497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:29.731676 14500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:29.731675 14492 server_base.cc:1034] running on GCE node
I20260621 05:18:29.732039 14492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:29.732318 14492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:29.733474 14492 hybrid_clock.cc:648] HybridClock initialized: now 1782019109733448 us; error 37 us; skew 500 ppm
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:29.735716 14492 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:18:29.736145 14492 webserver.cc:460] Webserver started at http://127.0.98.3:36311/ using document root <none> and password file <none>
I20260621 05:18:29.736366 14492 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:29.736430 14492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:29.736557 14492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:29.737507 14492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "ac57c882a8ed47d28eb931b8dcaa3936"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "ed2cbf54ff384d6c97460f2029037ccf"
server_key_iv: "ca9b16360103824e14115b98c793dec6"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.737841 14492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ac57c882a8ed47d28eb931b8dcaa3936"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "ed2cbf54ff384d6c97460f2029037ccf"
server_key_iv: "ca9b16360103824e14115b98c793dec6"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.739008 14492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:18:29.739663 14507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.739821 14492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.739916 14492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "ac57c882a8ed47d28eb931b8dcaa3936"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "ed2cbf54ff384d6c97460f2029037ccf"
server_key_iv: "ca9b16360103824e14115b98c793dec6"
server_key_version: "encryptionkey@0"
I20260621 05:18:29.739992 14492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:29.776767 14492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:29.778481 14492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:29.778640 14492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:29.778859 14492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:29.779191 14492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:29.779248 14492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.779282 14492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:29.779309 14492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:29.785014 14492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36967
I20260621 05:18:29.785058 14620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36967 every 8 connection(s)
I20260621 05:18:29.785329 14492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:18:29 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:29.792783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 14492
I20260621 05:18:29.792886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:18:29.793063 392 external_mini_cluster.cc:1428] Setting key c706957ed5126746bd6c250a032956e5
I20260621 05:18:29.793969 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.785931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49679 (local address 127.0.98.62:39703)
0621 05:18:29.785986 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.785988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.786200 (+ 212us) server_negotiation.cc:407] Connection header received
0621 05:18:29.786724 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.786725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.786737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.786806 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:29.787723 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.788183 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.789164 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.789363 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.789446 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.791358 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:29.791360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:29.791360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:29.791367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:29.792580 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.793063 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.793064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.793065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.793145 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.793397 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.793398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.793398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.793504 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:29.793576 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.793818 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.793888 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:18:29.794143 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.785909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:49679)
0621 05:18:29.786076 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.786107 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:29.786598 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:29.786877 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:29.786881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:29.787410 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:29.787572 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.787577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.788306 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.788308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.789063 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.789064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.789455 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.789456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.789576 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.789954 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:29.789961 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:29.791259 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:29.792678 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:29.792684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:29.792686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:29.792975 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:29.793229 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:29.793230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:29.793230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:29.793332 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:29.793571 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:29.793577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:29.793701 (+ 124us) client_negotiation.cc:769] Sending connection context
0621 05:18:29.793796 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:18:29.793934 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260621 05:18:29.794562 14621 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260621 05:18:29.794632 14621 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:29.794801 14621 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
I20260621 05:18:29.795130 14162 ts_manager.cc:194] Registered new tserver with Master: ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
I20260621 05:18:29.795465 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49679
I20260621 05:18:29.804497 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:18:29.811363 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.805763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.0.98.62:39703)
0621 05:18:29.805796 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.805797 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.805914 (+ 117us) server_negotiation.cc:407] Connection header received
0621 05:18:29.805950 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.805951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.805958 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.806036 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:29.806628 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.807067 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.807885 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.808063 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.808159 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.809290 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:29.809291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:29.809291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:29.809298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:29.810346 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.810666 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.810667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.810667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.810719 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:29.810901 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:29.810901 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:29.810902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:29.811028 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:29.811123 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.811211 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.811276 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:18:29.812805 14162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:18:29.814255 14162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:18:29.820505 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.816294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.0.1:34334)
0621 05:18:29.816579 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.816620 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:29.816682 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:29.816935 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:29.816938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:29.816942 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:29.816994 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.816995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.818268 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.818269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.819351 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.819353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.820203 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.820204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.820279 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.820290 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 05:18:29.820362 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:18:29.820416 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":89,"thread_start_us":42,"threads_started":1}
I20260621 05:18:29.820523 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.816487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.0.98.1:35049)
0621 05:18:29.816638 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.816643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.816673 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:18:29.816793 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.816796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.816814 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.816906 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:29.817117 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.818163 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.819437 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.820112 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.820187 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.820240 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.820388 (+ 148us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.820438 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260621 05:18:29.820658 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.816656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.0.98.3:36967)
0621 05:18:29.817027 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.817031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.817251 (+ 220us) server_negotiation.cc:407] Connection header received
0621 05:18:29.817284 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.817286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.817299 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.817358 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:29.817523 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.818157 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.819708 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.820184 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.820248 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.820299 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.820544 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.820591 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":86,"threads_started":1}
I20260621 05:18:29.820878 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.816327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34943 (local address 127.0.0.1:43044)
0621 05:18:29.816606 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.816620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:29.816682 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:29.817141 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:29.817142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:29.817144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:29.817184 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.817185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.818413 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.818414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.819594 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.819596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.820537 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.820538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.820719 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.820724 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:18:29.820767 (+ 43us) client_negotiation.cc:240] Negotiation successful
0621 05:18:29.820802 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":11,"thread_start_us":57,"threads_started":1}
I20260621 05:18:29.821355 14287 tablet_service.cc:1467] Processing CreateTablet for tablet e3050ffc536b407598b95c3236eebba6 (DEFAULT_TABLE table=test-table [id=7874524777cc4de98adeddca74da0d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:29.821712 14287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3050ffc536b407598b95c3236eebba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:29.823809 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.816430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.0.98.2:34943)
0621 05:18:29.816865 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.816870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.816901 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:18:29.816939 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.816941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.816955 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.817053 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:29.817316 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.818324 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.819701 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.820393 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.820455 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.820501 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.823708 (+ 3207us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.823751 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":86,"threads_started":1}
I20260621 05:18:29.820510 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.816480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.0.1:38676)
0621 05:18:29.817150 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.817159 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:29.817202 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:29.817423 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:29.817424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:29.817425 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:29.817454 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.817455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.818253 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.818254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.819594 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.819596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.820329 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.820330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.820383 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.820385 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:18:29.820427 (+ 42us) client_negotiation.cc:240] Negotiation successful
0621 05:18:29.820456 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":145,"thread_start_us":45,"threads_started":1}
I20260621 05:18:29.824415 14635 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Bootstrap starting.
I20260621 05:18:29.824573 14421 tablet_service.cc:1467] Processing CreateTablet for tablet e3050ffc536b407598b95c3236eebba6 (DEFAULT_TABLE table=test-table [id=7874524777cc4de98adeddca74da0d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:29.824674 14555 tablet_service.cc:1467] Processing CreateTablet for tablet e3050ffc536b407598b95c3236eebba6 (DEFAULT_TABLE table=test-table [id=7874524777cc4de98adeddca74da0d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:29.824883 14421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3050ffc536b407598b95c3236eebba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:29.824920 14555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3050ffc536b407598b95c3236eebba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:29.825070 14635 tablet_bootstrap.cc:654] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:29.825546 14635 log.cc:824] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:29.826249 14635 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: No bootstrap required, opened a new log
I20260621 05:18:29.826332 14635 ts_tablet_manager.cc:1397] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:18:29.827859 14635 raft_consensus.cc:348] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.828065 14635 raft_consensus.cc:374] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:29.828104 14635 raft_consensus.cc:729] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71198d3948694a8a97bed7e98ac29189, State: Initialized, Role: FOLLOWER
I20260621 05:18:29.828238 14635 consensus_queue.cc:260] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.827991 14637 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Bootstrap starting.
I20260621 05:18:29.828495 14635 ts_tablet_manager.cc:1428] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:18:29.828627 14353 heartbeater.cc:503] Master 127.0.98.62:39703 was elected leader, sending a full tablet report...
I20260621 05:18:29.828712 14638 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Bootstrap starting.
I20260621 05:18:29.829210 14637 tablet_bootstrap.cc:654] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:29.829661 14638 tablet_bootstrap.cc:654] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:29.829759 14637 log.cc:824] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:29.830190 14638 log.cc:824] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:29.830636 14637 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: No bootstrap required, opened a new log
I20260621 05:18:29.830708 14637 ts_tablet_manager.cc:1397] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:18:29.831005 14638 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: No bootstrap required, opened a new log
I20260621 05:18:29.831082 14638 ts_tablet_manager.cc:1397] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:18:29.832330 14637 raft_consensus.cc:348] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.832456 14637 raft_consensus.cc:374] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:29.832477 14637 raft_consensus.cc:729] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c241b29957d04ce0bca52ca5acb68487, State: Initialized, Role: FOLLOWER
I20260621 05:18:29.832612 14637 consensus_queue.cc:260] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.832885 14637 ts_tablet_manager.cc:1428] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:18:29.832907 14487 heartbeater.cc:503] Master 127.0.98.62:39703 was elected leader, sending a full tablet report...
I20260621 05:18:29.832794 14638 raft_consensus.cc:348] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.832940 14638 raft_consensus.cc:374] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:29.832983 14638 raft_consensus.cc:729] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac57c882a8ed47d28eb931b8dcaa3936, State: Initialized, Role: FOLLOWER
I20260621 05:18:29.833083 14638 consensus_queue.cc:260] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.833326 14638 ts_tablet_manager.cc:1428] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:18:29.833357 14621 heartbeater.cc:503] Master 127.0.98.62:39703 was elected leader, sending a full tablet report...
W20260621 05:18:29.870314 14488 tablet.cc:2369] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:18:29.927258 14642 raft_consensus.cc:482] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:29.927371 14642 raft_consensus.cc:504] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.927721 14642 leader_election.cc:288] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
I20260621 05:18:29.930675 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.927870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.98.2:33319)
0621 05:18:29.927930 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.927943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:29.927998 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:29.928188 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:29.928189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:29.928191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:29.928287 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.928306 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.929092 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.929094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.929815 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.929817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.930419 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.930420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.930505 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.930508 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:18:29.930562 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:18:29.930603 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260621 05:18:29.930701 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.928024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33319 (local address 127.0.98.1:35049)
0621 05:18:29.928074 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.928077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.928086 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:18:29.928114 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.928115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.928132 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.928198 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:29.928375 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.928851 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.929935 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.930316 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.930419 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.930430 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.930585 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.930636 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:18:29.931084 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.928042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44657 (local address 127.0.98.3:36967)
0621 05:18:29.928100 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.928102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.928216 (+ 114us) server_negotiation.cc:407] Connection header received
0621 05:18:29.928245 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.928246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.928258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.928312 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:29.928449 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.929137 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.930110 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.930570 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.930634 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.930643 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.930990 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.931027 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:18:29.931093 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.927986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.98.2:44657)
0621 05:18:29.928132 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.928141 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:29.928194 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:29.928328 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:29.928329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:29.928330 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:29.928389 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.928390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.929248 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.929249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.930015 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:29.930017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.930676 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.930677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:29.930922 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.930926 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:18:29.930984 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:18:29.931020 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260621 05:18:29.931072 14307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3050ffc536b407598b95c3236eebba6" candidate_uuid: "c241b29957d04ce0bca52ca5acb68487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71198d3948694a8a97bed7e98ac29189" is_pre_election: true
I20260621 05:18:29.931250 14307 raft_consensus.cc:2454] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c241b29957d04ce0bca52ca5acb68487 in term 0.
I20260621 05:18:29.931506 14376 leader_election.cc:302] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71198d3948694a8a97bed7e98ac29189, c241b29957d04ce0bca52ca5acb68487; no voters:
I20260621 05:18:29.931695 14642 raft_consensus.cc:2781] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:18:29.931679 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3050ffc536b407598b95c3236eebba6" candidate_uuid: "c241b29957d04ce0bca52ca5acb68487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" is_pre_election: true
I20260621 05:18:29.931766 14642 raft_consensus.cc:482] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:18:29.931792 14642 raft_consensus.cc:3037] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:29.931834 14575 raft_consensus.cc:2454] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c241b29957d04ce0bca52ca5acb68487 in term 0.
I20260621 05:18:29.932523 14642 raft_consensus.cc:504] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.932636 14642 leader_election.cc:288] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [CANDIDATE]: Term 1 election: Requested vote from peers 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
I20260621 05:18:29.932842 14307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3050ffc536b407598b95c3236eebba6" candidate_uuid: "c241b29957d04ce0bca52ca5acb68487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71198d3948694a8a97bed7e98ac29189"
I20260621 05:18:29.932906 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3050ffc536b407598b95c3236eebba6" candidate_uuid: "c241b29957d04ce0bca52ca5acb68487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac57c882a8ed47d28eb931b8dcaa3936"
I20260621 05:18:29.932940 14307 raft_consensus.cc:3037] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:29.933000 14575 raft_consensus.cc:3037] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:29.933624 14307 raft_consensus.cc:2454] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c241b29957d04ce0bca52ca5acb68487 in term 1.
I20260621 05:18:29.933632 14575 raft_consensus.cc:2454] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c241b29957d04ce0bca52ca5acb68487 in term 1.
I20260621 05:18:29.933791 14376 leader_election.cc:302] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac57c882a8ed47d28eb931b8dcaa3936, c241b29957d04ce0bca52ca5acb68487; no voters:
I20260621 05:18:29.933893 14642 raft_consensus.cc:2781] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:29.934041 14642 raft_consensus.cc:686] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 1 LEADER]: Becoming Leader. State: Replica: c241b29957d04ce0bca52ca5acb68487, State: Running, Role: LEADER
I20260621 05:18:29.934123 14642 consensus_queue.cc:237] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:29.934846 14162 catalog_manager.cc:5495] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 reported cstate change: term changed from 0 to 1, leader changed from <none> to c241b29957d04ce0bca52ca5acb68487 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c241b29957d04ce0bca52ca5acb68487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } health_report { overall_health: HEALTHY } } }
I20260621 05:18:29.963251 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.961232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.0.98.2:34943)
0621 05:18:29.961271 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:29.961273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:29.961287 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:18:29.961346 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:29.961346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:29.961371 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:29.961421 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:18:29.961509 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.961964 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.962664 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:29.962847 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:29.962899 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:29.963020 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:18:29.963106 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:18:29.963159 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:29.963190 (+ 31us) server_negotiation.cc:299] Negotiation successful
0621 05:18:29.963204 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:18:29.964931 14307 raft_consensus.cc:1261] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Refusing update from remote peer c241b29957d04ce0bca52ca5acb68487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:29.965140 14575 raft_consensus.cc:1261] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Refusing update from remote peer c241b29957d04ce0bca52ca5acb68487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:29.965258 14645 consensus_queue.cc:1035] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:29.965435 14642 consensus_queue.cc:1035] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:29.967717 14648 mvcc.cc:205] Tried to move back new op lower bound from 7299150274413441024 to 7299150274290724864. Current Snapshot: MvccSnapshot[applied={T|T < 7299150274413441024}]
I20260621 05:18:29.968511 14649 mvcc.cc:205] Tried to move back new op lower bound from 7299150274413441024 to 7299150274290724864. Current Snapshot: MvccSnapshot[applied={T|T < 7299150274413441024 or (T in {7299150274413441024})}]
I20260621 05:18:29.968621 14650 mvcc.cc:205] Tried to move back new op lower bound from 7299150274413441024 to 7299150274290724864. Current Snapshot: MvccSnapshot[applied={T|T < 7299150274413441024 or (T in {7299150274413441024})}]
I20260621 05:18:29.970458 14162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:18:29.970535 14162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:18:29.971191 14162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Sending DeleteTablet for 3 replicas of tablet e3050ffc536b407598b95c3236eebba6
I20260621 05:18:29.971530 14287 tablet_service.cc:1514] Processing DeleteTablet for tablet e3050ffc536b407598b95c3236eebba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34334
I20260621 05:18:29.971554 14421 tablet_service.cc:1514] Processing DeleteTablet for tablet e3050ffc536b407598b95c3236eebba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43044
I20260621 05:18:29.971604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 14224
I20260621 05:18:29.972167 14659 tablet_replica.cc:330] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: stopping tablet replica
I20260621 05:18:29.972265 14659 raft_consensus.cc:2229] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:18:29.972380 14659 raft_consensus.cc:2258] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:29.972779 14659 ts_tablet_manager.cc:1905] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:29.974643 14659 ts_tablet_manager.cc:1918] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:29.974712 14659 log.cc:1196] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/e3050ffc536b407598b95c3236eebba6
I20260621 05:18:29.974900 14555 tablet_service.cc:1514] Processing DeleteTablet for tablet e3050ffc536b407598b95c3236eebba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38676
I20260621 05:18:29.974970 14659 ts_tablet_manager.cc:1939] T e3050ffc536b407598b95c3236eebba6 P c241b29957d04ce0bca52ca5acb68487: Deleting consensus metadata
I20260621 05:18:29.975324 14147 catalog_manager.cc:4841] TS c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943): tablet e3050ffc536b407598b95c3236eebba6 (table test-table [id=7874524777cc4de98adeddca74da0d65]) successfully deleted
W20260621 05:18:29.977910 14148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35049 (error 108)
I20260621 05:18:29.978355 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.978234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.0.1:34344)
0621 05:18:29.978288 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.978322 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 05:18:29.978397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 14358
W20260621 05:18:29.979763 14148 catalog_manager.cc:4584] TS 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3050ffc536b407598b95c3236eebba6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35049: connect: Connection refused (error 111)
I20260621 05:18:29.982797 14660 tablet_replica.cc:330] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: stopping tablet replica
I20260621 05:18:29.982909 14660 raft_consensus.cc:2229] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:18:29.983004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 14492
I20260621 05:18:29.983026 14660 raft_consensus.cc:2258] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:29.987478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 14130
I20260621 05:18:29.987507 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:29.987400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.0.1:38686)
0621 05:18:29.987446 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:29.987478 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:36967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 05:18:29.992550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39703
--webserver_interface=127.0.98.62
--webserver_port=39705
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:30.070503 14662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:30.070669 14662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:30.070688 14662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:30.071738 14662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:30.071771 14662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:30.071785 14662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:30.071798 14662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:30.071810 14662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:30.073143 14662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:30.073297 14662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:30.073436 14662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:30.081099 14671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:30.081069 14668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:30.081107 14669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:30.081154 14662 server_base.cc:1034] running on GCE node
I20260621 05:18:30.081446 14662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:30.081771 14662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:30.082948 14662 hybrid_clock.cc:648] HybridClock initialized: now 1782019110082934 us; error 46 us; skew 500 ppm
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:30.085155 14662 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:18:30.085577 14662 webserver.cc:460] Webserver started at http://127.0.98.62:39705/ using document root <none> and password file <none>
I20260621 05:18:30.085803 14662 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:30.085876 14662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:30.087162 14662 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:30.087802 14678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:30.087975 14662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:30.088064 14662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "a2bdc27fffeb4ea1a49e5eb73681356e"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "3fef888d309d1d76b2c7f782bf407f81"
server_key_iv: "7291ab587cbc114ec03e04d71f21c3b3"
server_key_version: "encryptionkey@0"
I20260621 05:18:30.088436 14662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:30.096813 14662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:30.098413 14662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:30.098564 14662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:30.102277 14662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39703
I20260621 05:18:30.102319 14730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39703 every 8 connection(s)
I20260621 05:18:30.102684 14662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:18:30.104110 14731 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:18:30.104683 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Bootstrap starting.
I20260621 05:18:30.106842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 14662
I20260621 05:18:30.107132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35049
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39703
--webserver_port=33233
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:30.112890 14731 log.cc:824] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:30.114425 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:18:30.114660 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Bootstrap complete.
I20260621 05:18:30.116786 14731 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:30.117013 14731 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bdc27fffeb4ea1a49e5eb73681356e, State: Initialized, Role: FOLLOWER
I20260621 05:18:30.117224 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:30.117308 14731 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:18:30.117372 14731 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:18:30.117412 14731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:18:30.118251 14731 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:30.118355 14731 leader_election.cc:302] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2bdc27fffeb4ea1a49e5eb73681356e; no voters:
I20260621 05:18:30.118525 14731 leader_election.cc:288] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:18:30.118559 14735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:18:30.118659 14735 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 2 LEADER]: Becoming Leader. State: Replica: a2bdc27fffeb4ea1a49e5eb73681356e, State: Running, Role: LEADER
I20260621 05:18:30.118733 14735 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:30.118755 14731 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:30.118973 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } } }
I20260621 05:18:30.119097 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:30.118970 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2bdc27fffeb4ea1a49e5eb73681356e. Latest consensus state: current_term: 2 leader_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } } }
I20260621 05:18:30.119185 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:30.119729 14742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:18:30.120357 14742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:18:30.120443 14742 catalog_manager.cc:1259] Loaded cluster ID: c33d6ec7334042d29abef11b347ccffb
I20260621 05:18:30.120472 14742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:18:30.120847 14742 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 05:18:30.125792 14742 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 05:18:30.127972 14742 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:18:30.128135 14742 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Loaded TSK: 0
I20260621 05:18:30.128320 14742 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:18:30.187948 14733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:30.188130 14733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:30.188151 14733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:30.189196 14733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:30.189234 14733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:30.189256 14733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:18:30.190677 14733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33233
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:30.190891 14733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:30.191100 14733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:30.195674 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:30.195752 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:30.195703 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:30.196044 14733 server_base.cc:1034] running on GCE node
I20260621 05:18:30.196226 14733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:30.196481 14733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:30.197631 14733 hybrid_clock.cc:648] HybridClock initialized: now 1782019110197621 us; error 26 us; skew 500 ppm
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:30.199945 14733 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:18:30.200392 14733 webserver.cc:460] Webserver started at http://127.0.98.1:33233/ using document root <none> and password file <none>
I20260621 05:18:30.200629 14733 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:30.200721 14733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:30.202045 14733 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:30.202807 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:30.203028 14733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:30.203117 14733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "71198d3948694a8a97bed7e98ac29189"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "bde24245807fd77a3e49cceb1b9d657b"
server_key_iv: "4453ccc6117bd0a3b62e2e87397c120b"
server_key_version: "encryptionkey@0"
I20260621 05:18:30.203415 14733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:30.228477 14733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:30.230711 14733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:30.230870 14733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:30.231122 14733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:30.231570 14775 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:18:30.233450 14733 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:18:30.233515 14733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:18:30.233560 14733 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:18:30.234202 14733 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:18:30.234256 14733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:30.234309 14775 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Bootstrap starting.
I20260621 05:18:30.240955 14733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35049
I20260621 05:18:30.241070 14882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35049 every 8 connection(s)
I20260621 05:18:30.241343 14733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:18:30.242043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 14733
I20260621 05:18:30.242339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34943
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39703
--webserver_port=36183
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:30.244335 14775 log.cc:824] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:30.246616 14775 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:18:30.246901 14775 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Bootstrap complete.
I20260621 05:18:30.248059 14775 ts_tablet_manager.cc:1397] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:30.249629 14775 raft_consensus.cc:348] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:30.250008 14775 raft_consensus.cc:729] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71198d3948694a8a97bed7e98ac29189, State: Initialized, Role: FOLLOWER
I20260621 05:18:30.250186 14775 consensus_queue.cc:260] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:30.250466 14775 ts_tablet_manager.cc:1428] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260621 05:18:30.252931 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.242201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35921 (local address 127.0.98.62:39703)
0621 05:18:30.242382 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.242390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.242652 (+ 262us) server_negotiation.cc:407] Connection header received
0621 05:18:30.243256 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.243261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.243300 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.243419 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:30.244297 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.245561 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.246606 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.247230 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.247338 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.249494 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:30.249498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:30.249498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:30.249509 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:30.251221 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.251678 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.251681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.251682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.251772 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.252037 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.252038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.252038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.252247 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:30.252332 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.252628 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.252769 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260621 05:18:30.252930 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.242126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:35921)
0621 05:18:30.242518 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.242564 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:30.243042 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:30.243460 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:30.243464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:30.243865 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:30.244139 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.244146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.245674 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.245675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.246483 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.246491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.247331 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.247332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.247428 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.247987 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:30.247999 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:30.249392 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:30.251306 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.251312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.251314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.251531 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.251845 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.251846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.251846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.251960 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.252324 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:30.252329 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:30.252458 (+ 129us) client_negotiation.cc:769] Sending connection context
0621 05:18:30.252565 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:18:30.252742 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":92,"threads_started":1}
I20260621 05:18:30.253692 14883 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260621 05:18:30.253788 14883 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:30.253944 14883 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
I20260621 05:18:30.254606 14695 ts_manager.cc:194] Registered new tserver with Master: 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049)
I20260621 05:18:30.255580 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35921
I20260621 05:18:30.256402 14883 heartbeater.cc:503] Master 127.0.98.62:39703 was elected leader, sending a full tablet report...
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260621 05:18:30.262949 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.255120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.0.1:34352)
0621 05:18:30.255283 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.255328 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:30.255390 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:30.255966 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:30.255969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:30.256276 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:30.256324 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.256326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.257350 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.257351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.258221 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.258223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.258616 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.258619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.258684 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.259060 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:30.259064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:30.260301 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:30.262007 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.262013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.262014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.262193 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.262390 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.262391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.262391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.262440 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.262727 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:30.262732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:30.262776 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:18:30.262828 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:18:30.262896 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260621 05:18:30.263028 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.255316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.0.98.1:35049)
0621 05:18:30.255772 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.255776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.255786 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:18:30.255814 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.255816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.255829 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.255905 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:30.256409 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.257284 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.258295 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.258541 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.258600 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.260372 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:30.260378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:30.260378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:30.260384 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:30.261922 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.262265 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.262267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.262268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.262325 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.262496 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.262497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.262497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.262654 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:30.262753 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.262868 (+ 115us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.262944 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":62,"threads_started":1}
I20260621 05:18:30.263271 14799 tablet_service.cc:1514] Processing DeleteTablet for tablet e3050ffc536b407598b95c3236eebba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34352
I20260621 05:18:30.263793 14892 tablet_replica.cc:330] stopping tablet replica
I20260621 05:18:30.263899 14892 raft_consensus.cc:2229] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:18:30.263976 14892 raft_consensus.cc:2258] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:30.264211 14892 ts_tablet_manager.cc:1905] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:30.266044 14892 ts_tablet_manager.cc:1918] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:30.266129 14892 log.cc:1196] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/e3050ffc536b407598b95c3236eebba6
I20260621 05:18:30.266310 14892 ts_tablet_manager.cc:1939] T e3050ffc536b407598b95c3236eebba6 P 71198d3948694a8a97bed7e98ac29189: Deleting consensus metadata
I20260621 05:18:30.266755 14681 catalog_manager.cc:4841] TS 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049): tablet e3050ffc536b407598b95c3236eebba6 (table test-table [id=7874524777cc4de98adeddca74da0d65]) successfully deleted
W20260621 05:18:30.326874 14887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:30.327059 14887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:30.327080 14887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:30.328186 14887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:30.328225 14887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:30.328276 14887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:18:30.329702 14887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36183
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:30.329906 14887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:30.330096 14887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:30.334707 14898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:30.334659 14900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:30.334867 14887 server_base.cc:1034] running on GCE node
W20260621 05:18:30.334659 14897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:30.335108 14887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:30.335283 14887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:30.336414 14887 hybrid_clock.cc:648] HybridClock initialized: now 1782019110336401 us; error 27 us; skew 500 ppm
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:30.338579 14887 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:18:30.339047 14887 webserver.cc:460] Webserver started at http://127.0.98.2:36183/ using document root <none> and password file <none>
I20260621 05:18:30.339269 14887 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:30.339319 14887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:30.340648 14887 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:30.341167 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:30.341329 14887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:30.341403 14887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "c241b29957d04ce0bca52ca5acb68487"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "7a270fe2e5dbc3bff83e028840c76203"
server_key_iv: "ddc07c8d18e87702eaebc3825a4f51c0"
server_key_version: "encryptionkey@0"
I20260621 05:18:30.341750 14887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:30.355023 14887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:30.356666 14887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:30.356822 14887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:30.357065 14887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:30.357345 14887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:30.357393 14887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:30.357422 14887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:30.357446 14887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:30.363188 14887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34943
I20260621 05:18:30.363221 15020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34943 every 8 connection(s)
I20260621 05:18:30.363516 14887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:30.370784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 14887
I20260621 05:18:30.371037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36967
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39703
--webserver_port=36311
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:30.371831 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.364194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40061 (local address 127.0.98.62:39703)
0621 05:18:30.364258 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.364260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.364470 (+ 210us) server_negotiation.cc:407] Connection header received
0621 05:18:30.364892 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.364893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.364905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.364957 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:30.365777 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.366222 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.367045 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.367235 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.367288 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.369310 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:30.369311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:30.369311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:30.369317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:30.370345 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.370772 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.370774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.370774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.370831 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.371099 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.371100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.371101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.371256 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:30.371322 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.371577 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.371691 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:18:30.371853 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.364175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:40061)
0621 05:18:30.364353 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.364390 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:30.364817 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:30.365017 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:30.365019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:30.365552 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:30.365711 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.365715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.366292 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.366293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.366982 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.366984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.367305 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.367306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.367420 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.367862 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:30.367869 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:30.369235 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:30.370408 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.370413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.370414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.370604 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.370928 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.370928 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.370929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.371033 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.371325 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:30.371329 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:30.371433 (+ 104us) client_negotiation.cc:769] Sending connection context
0621 05:18:30.371518 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 05:18:30.371696 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260621 05:18:30.372319 15021 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260621 05:18:30.372421 15021 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:30.372609 15021 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
I20260621 05:18:30.372993 14695 ts_manager.cc:194] Registered new tserver with Master: c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943)
I20260621 05:18:30.373394 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40061
W20260621 05:18:30.454312 15024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:30.454497 15024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:30.454519 15024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:30.455556 15024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:30.455602 15024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:30.455677 15024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:18:30.457144 15024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36311
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:30.457393 15024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:30.457613 15024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:30.462380 15029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:30.462343 15030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:30.462401 15024 server_base.cc:1034] running on GCE node
W20260621 05:18:30.462356 15032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:30.462738 15024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:30.462971 15024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:30.464128 15024 hybrid_clock.cc:648] HybridClock initialized: now 1782019110464110 us; error 47 us; skew 500 ppm
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:30.466320 15024 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:18:30.466791 15024 webserver.cc:460] Webserver started at http://127.0.98.3:36311/ using document root <none> and password file <none>
I20260621 05:18:30.467011 15024 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:30.467096 15024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:30.468410 15024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:18:30.469094 15039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:30.469236 15024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:18:30.469309 15024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "ac57c882a8ed47d28eb931b8dcaa3936"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "ed2cbf54ff384d6c97460f2029037ccf"
server_key_iv: "ca9b16360103824e14115b98c793dec6"
server_key_version: "encryptionkey@0"
I20260621 05:18:30.469590 15024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:30.480008 15024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:30.481415 15024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:30.481506 15024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:30.481679 15024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:30.482007 15046 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:18:30.483263 15024 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:18:30.483299 15024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:30.483318 15024 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:18:30.483883 15024 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:18:30.483914 15024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:30.483999 15046 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Bootstrap starting.
I20260621 05:18:30.489275 15024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36967
I20260621 05:18:30.489368 15153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36967 every 8 connection(s)
I20260621 05:18:30.489641 15024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:18:30.494398 15046 log.cc:824] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Log is configured to *not* fsync() on all Append() calls
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:30.496249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 15024
I20260621 05:18:30.496997 15046 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:18:30.497329 15046 tablet_bootstrap.cc:492] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Bootstrap complete.
I20260621 05:18:30.498452 15046 ts_tablet_manager.cc:1397] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260621 05:18:30.499289 15046 raft_consensus.cc:348] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:30.499519 15046 raft_consensus.cc:729] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac57c882a8ed47d28eb931b8dcaa3936, State: Initialized, Role: FOLLOWER
I20260621 05:18:30.499574 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.490354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43693 (local address 127.0.98.62:39703)
0621 05:18:30.490429 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.490431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.490596 (+ 165us) server_negotiation.cc:407] Connection header received
0621 05:18:30.491087 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.491089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.491101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.491177 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:30.491898 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.492739 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.493674 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.493974 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.494051 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.496602 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:30.496604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:30.496604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:30.496611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:30.498219 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.498636 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.498638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.498639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.498697 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.498951 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.498952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.498952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.499061 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:30.499139 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.499402 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.499481 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:18:30.499677 15046 consensus_queue.cc:260] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } }
I20260621 05:18:30.499809 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.490319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:43693)
0621 05:18:30.490483 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.490515 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:30.490975 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:30.491250 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:30.491254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:30.491549 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:30.491806 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.491811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.492846 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.492847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.493581 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.493583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.494072 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.494073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.494176 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.494795 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:30.494804 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:30.496477 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:30.498303 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.498308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.498310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.498538 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.498777 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.498778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.498778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.498881 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.499142 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:30.499148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:30.499272 (+ 124us) client_negotiation.cc:769] Sending connection context
0621 05:18:30.499369 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 05:18:30.499519 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260621 05:18:30.499950 15046 ts_tablet_manager.cc:1428] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260621 05:18:30.500247 15154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260621 05:18:30.500336 15154 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:30.500501 15154 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
I20260621 05:18:30.501060 14695 ts_manager.cc:194] Registered new tserver with Master: ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
I20260621 05:18:30.501582 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43693
I20260621 05:18:30.502318 15154 heartbeater.cc:503] Master 127.0.98.62:39703 was elected leader, sending a full tablet report...
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260621 05:18:30.506799 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.496736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.0.98.62:39703)
0621 05:18:30.499662 (+ 2926us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.499664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.499672 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:18:30.499691 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.499692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.499700 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.499752 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:30.500480 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.501073 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.502145 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.502403 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.502471 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.503790 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:30.503791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:30.503791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:30.503797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:30.505341 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.505703 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.505703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.505704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.505778 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.505992 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.505993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.505994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.506172 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:30.506356 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.506635 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.506706 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2913,"thread_start_us":47,"threads_started":1}
I20260621 05:18:30.507516 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:18:30.509163 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.501333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.0.1:38698)
0621 05:18:30.501389 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.501399 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:30.501435 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:30.501822 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:30.501823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:30.501974 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:30.502022 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.502023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.503126 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.503127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.503616 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.503618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.504242 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.504245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.504305 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.504615 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:30.504619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:30.506165 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:30.508234 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.508234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.508235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.508422 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.508631 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:30.508631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:30.508632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:30.508689 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:30.508930 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:30.508931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:30.508968 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 05:18:30.509042 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:18:30.509109 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:18:30.509341 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.501516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.98.3:36967)
0621 05:18:30.501686 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.501690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.501700 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:18:30.501730 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.501732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.501746 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.501810 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:30.502103 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.503059 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.503751 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.504173 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.504236 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.506251 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:30.506258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:30.506258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:30.506265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:30.508170 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.508490 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.508492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.508492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.508558 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:30.508741 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:30.508741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:30.508741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:30.508850 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:30.508940 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.509099 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.509185 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260621 05:18:30.509613 15088 tablet_service.cc:1514] Processing DeleteTablet for tablet e3050ffc536b407598b95c3236eebba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38698
I20260621 05:18:30.510022 15163 tablet_replica.cc:330] stopping tablet replica
I20260621 05:18:30.510133 15163 raft_consensus.cc:2229] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:18:30.510210 15163 raft_consensus.cc:2258] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:30.510504 15163 ts_tablet_manager.cc:1905] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:30.510707 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.507863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.0.98.62:39703)
0621 05:18:30.507899 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.507901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.507971 (+ 70us) server_negotiation.cc:407] Connection header received
0621 05:18:30.508042 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.508043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.508067 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.508118 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:18:30.508226 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.508839 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.509820 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.510091 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.510188 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.510411 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:18:30.510510 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:18:30.510605 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.510624 (+ 19us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.510640 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:18:30.512669 15163 ts_tablet_manager.cc:1918] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:30.512743 15163 log.cc:1196] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/e3050ffc536b407598b95c3236eebba6
I20260621 05:18:30.512970 15163 ts_tablet_manager.cc:1939] T e3050ffc536b407598b95c3236eebba6 P ac57c882a8ed47d28eb931b8dcaa3936: Deleting consensus metadata
I20260621 05:18:30.510924 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:18:30.513319 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:18:30.513477 14681 catalog_manager.cc:4841] TS ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967): tablet e3050ffc536b407598b95c3236eebba6 (table test-table [id=7874524777cc4de98adeddca74da0d65]) successfully deleted
I20260621 05:18:30.515091 15088 tablet_service.cc:1467] Processing CreateTablet for tablet c878f538cc2b42eb8cfe2937f57c834b (DEFAULT_TABLE table=test-table [id=44ffdab725034aa18743e4c0b86c3c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:30.515117 14799 tablet_service.cc:1467] Processing CreateTablet for tablet c878f538cc2b42eb8cfe2937f57c834b (DEFAULT_TABLE table=test-table [id=44ffdab725034aa18743e4c0b86c3c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:30.515273 15088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c878f538cc2b42eb8cfe2937f57c834b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:30.515280 14799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c878f538cc2b42eb8cfe2937f57c834b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:30.516490 14775 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Bootstrap starting.
I20260621 05:18:30.516602 15046 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: Bootstrap starting.
I20260621 05:18:30.516774 14775 tablet_bootstrap.cc:654] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:30.516875 15046 tablet_bootstrap.cc:654] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:30.517447 15046 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: No bootstrap required, opened a new log
I20260621 05:18:30.517504 15046 ts_tablet_manager.cc:1397] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:30.517786 15046 raft_consensus.cc:348] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.517856 15046 raft_consensus.cc:374] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:30.517894 15046 raft_consensus.cc:729] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac57c882a8ed47d28eb931b8dcaa3936, State: Initialized, Role: FOLLOWER
I20260621 05:18:30.517966 15046 consensus_queue.cc:260] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.518092 15046 ts_tablet_manager.cc:1428] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:30.519188 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.514909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34943 (local address 127.0.0.1:43070)
0621 05:18:30.514948 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.514958 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:30.515000 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:30.515408 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:30.515409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:30.515410 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:30.515450 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.515451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.516735 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.516736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.517870 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.517873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.518937 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.518938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.519006 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.519025 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 05:18:30.519072 (+ 47us) client_negotiation.cc:240] Negotiation successful
0621 05:18:30.519120 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260621 05:18:30.519330 14775 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: No bootstrap required, opened a new log
I20260621 05:18:30.519379 14775 ts_tablet_manager.cc:1397] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260621 05:18:30.519488 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.515019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.0.98.2:34943)
0621 05:18:30.515217 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.515222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.515232 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:18:30.515269 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.515271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.515288 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.515395 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:30.515521 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.516608 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.518007 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.518836 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.518911 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.518965 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.519359 (+ 394us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.519404 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
I20260621 05:18:30.519518 14775 raft_consensus.cc:348] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.519589 14775 raft_consensus.cc:374] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:30.519701 14775 raft_consensus.cc:729] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71198d3948694a8a97bed7e98ac29189, State: Initialized, Role: FOLLOWER
I20260621 05:18:30.519754 14775 consensus_queue.cc:260] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.519853 14775 ts_tablet_manager.cc:1428] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:30.520337 14955 tablet_service.cc:1467] Processing CreateTablet for tablet c878f538cc2b42eb8cfe2937f57c834b (DEFAULT_TABLE table=test-table [id=44ffdab725034aa18743e4c0b86c3c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:30.520668 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c878f538cc2b42eb8cfe2937f57c834b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:30.523303 15167 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Bootstrap starting.
I20260621 05:18:30.523945 15167 tablet_bootstrap.cc:654] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:30.524470 15167 log.cc:824] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:30.525161 15167 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: No bootstrap required, opened a new log
I20260621 05:18:30.525213 15167 ts_tablet_manager.cc:1397] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:18:30.526446 15167 raft_consensus.cc:348] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.526571 15167 raft_consensus.cc:374] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:30.526628 15167 raft_consensus.cc:729] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c241b29957d04ce0bca52ca5acb68487, State: Initialized, Role: FOLLOWER
I20260621 05:18:30.526758 15167 consensus_queue.cc:260] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.527011 15167 ts_tablet_manager.cc:1428] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:18:30.527035 15021 heartbeater.cc:503] Master 127.0.98.62:39703 was elected leader, sending a full tablet report...
W20260621 05:18:30.614058 15022 tablet.cc:2369] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:18:30.740276 15155 tablet.cc:2369] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:18:30.742250 14884 tablet.cc:2369] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:18:30.802682 15169 raft_consensus.cc:482] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:30.802811 15169 raft_consensus.cc:504] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.803261 15169 leader_election.cc:288] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
I20260621 05:18:30.807471 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.803532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46811 (local address 127.0.98.1:35049)
0621 05:18:30.803705 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.803707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.803716 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:18:30.803746 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.803747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.803772 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.803873 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:30.804065 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.804884 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.806451 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.807086 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.807160 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.807240 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.807356 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.807397 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260621 05:18:30.807468 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.803689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56873 (local address 127.0.98.3:36967)
0621 05:18:30.803742 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.803744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.803849 (+ 105us) server_negotiation.cc:407] Connection header received
0621 05:18:30.803930 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.803931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.803955 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.804000 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:30.804150 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.804886 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.806438 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.807079 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.807160 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.807240 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.807356 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.807398 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:18:30.807482 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.803468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.98.2:46811)
0621 05:18:30.803538 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.803552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:30.803613 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:30.803890 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:30.803891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:30.803893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:30.803973 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.803974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.804967 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.804968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.806332 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.806335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.807175 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.807176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.807272 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.807274 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:18:30.807339 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 05:18:30.807386 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":40}
I20260621 05:18:30.807488 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.803606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.98.2:56873)
0621 05:18:30.803768 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.803785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:30.803839 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:30.804023 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:30.804023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:30.804025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:30.804082 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.804083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.804967 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.804968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.806332 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:30.806335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.807175 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.807176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:30.807269 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.807274 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:18:30.807339 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 05:18:30.807386 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":101,"thread_start_us":106,"threads_started":1}
I20260621 05:18:30.807830 14837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c878f538cc2b42eb8cfe2937f57c834b" candidate_uuid: "c241b29957d04ce0bca52ca5acb68487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71198d3948694a8a97bed7e98ac29189" is_pre_election: true
I20260621 05:18:30.808008 14837 raft_consensus.cc:2454] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c241b29957d04ce0bca52ca5acb68487 in term 0.
I20260621 05:18:30.807961 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c878f538cc2b42eb8cfe2937f57c834b" candidate_uuid: "c241b29957d04ce0bca52ca5acb68487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" is_pre_election: true
I20260621 05:18:30.808111 15108 raft_consensus.cc:2454] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c241b29957d04ce0bca52ca5acb68487 in term 0.
I20260621 05:18:30.808264 14910 leader_election.cc:302] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71198d3948694a8a97bed7e98ac29189, c241b29957d04ce0bca52ca5acb68487; no voters:
I20260621 05:18:30.808408 15169 raft_consensus.cc:2781] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:18:30.808465 15169 raft_consensus.cc:482] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:18:30.808506 15169 raft_consensus.cc:3037] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:30.809219 15169 raft_consensus.cc:504] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.809377 15169 leader_election.cc:288] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [CANDIDATE]: Term 1 election: Requested vote from peers 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
I20260621 05:18:30.809536 14837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c878f538cc2b42eb8cfe2937f57c834b" candidate_uuid: "c241b29957d04ce0bca52ca5acb68487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71198d3948694a8a97bed7e98ac29189"
I20260621 05:18:30.809549 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c878f538cc2b42eb8cfe2937f57c834b" candidate_uuid: "c241b29957d04ce0bca52ca5acb68487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac57c882a8ed47d28eb931b8dcaa3936"
I20260621 05:18:30.809655 14837 raft_consensus.cc:3037] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:30.809652 15108 raft_consensus.cc:3037] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:30.810304 14837 raft_consensus.cc:2454] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c241b29957d04ce0bca52ca5acb68487 in term 1.
I20260621 05:18:30.810344 15108 raft_consensus.cc:2454] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c241b29957d04ce0bca52ca5acb68487 in term 1.
I20260621 05:18:30.810467 14910 leader_election.cc:302] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71198d3948694a8a97bed7e98ac29189, c241b29957d04ce0bca52ca5acb68487; no voters:
I20260621 05:18:30.810576 15169 raft_consensus.cc:2781] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:30.810731 15169 raft_consensus.cc:686] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 1 LEADER]: Becoming Leader. State: Replica: c241b29957d04ce0bca52ca5acb68487, State: Running, Role: LEADER
I20260621 05:18:30.810853 15169 consensus_queue.cc:237] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:30.811647 14695 catalog_manager.cc:5495] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 reported cstate change: term changed from 0 to 1, leader changed from <none> to c241b29957d04ce0bca52ca5acb68487 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c241b29957d04ce0bca52ca5acb68487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } health_report { overall_health: UNKNOWN } } }
I20260621 05:18:30.870960 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.868777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.0.98.2:34943)
0621 05:18:30.868831 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:30.868835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:30.868846 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:18:30.868878 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:30.868878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:30.868897 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:30.868953 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:18:30.869055 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.869496 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.870308 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:30.870496 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:30.870547 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:30.870702 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:18:30.870793 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:18:30.870847 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:30.870876 (+ 29us) server_negotiation.cc:299] Negotiation successful
0621 05:18:30.870892 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:18:30.872761 14837 raft_consensus.cc:1261] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Refusing update from remote peer c241b29957d04ce0bca52ca5acb68487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:30.872804 15108 raft_consensus.cc:1261] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Refusing update from remote peer c241b29957d04ce0bca52ca5acb68487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:30.873055 15169 consensus_queue.cc:1035] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:30.873222 15172 consensus_queue.cc:1035] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:30.875237 15175 mvcc.cc:205] Tried to move back new op lower bound from 7299150278132199424 to 7299150277881716736. Current Snapshot: MvccSnapshot[applied={T|T < 7299150278132199424}]
I20260621 05:18:30.875763 15176 mvcc.cc:205] Tried to move back new op lower bound from 7299150278132199424 to 7299150277881716736. Current Snapshot: MvccSnapshot[applied={T|T < 7299150278132199424}]
I20260621 05:18:30.877663 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:18:30.877750 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:18:30.878443 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Sending DeleteTablet for 3 replicas of tablet c878f538cc2b42eb8cfe2937f57c834b
I20260621 05:18:30.878733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 14733
I20260621 05:18:30.878726 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet c878f538cc2b42eb8cfe2937f57c834b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43070
I20260621 05:18:30.880570 15183 tablet_replica.cc:330] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: stopping tablet replica
I20260621 05:18:30.880604 15088 tablet_service.cc:1514] Processing DeleteTablet for tablet c878f538cc2b42eb8cfe2937f57c834b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38698
I20260621 05:18:30.880679 15183 raft_consensus.cc:2229] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:18:30.880700 15163 tablet_replica.cc:330] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: stopping tablet replica
I20260621 05:18:30.880755 15163 raft_consensus.cc:2229] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:18:30.880810 15183 raft_consensus.cc:2258] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:30.880822 15163 raft_consensus.cc:2258] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:30.881191 15183 ts_tablet_manager.cc:1905] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:30.883158 15183 ts_tablet_manager.cc:1918] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:30.883234 15183 log.cc:1196] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/c878f538cc2b42eb8cfe2937f57c834b
I20260621 05:18:30.883479 15183 ts_tablet_manager.cc:1939] T c878f538cc2b42eb8cfe2937f57c834b P c241b29957d04ce0bca52ca5acb68487: Deleting consensus metadata
I20260621 05:18:30.883877 14680 catalog_manager.cc:4841] TS c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943): tablet c878f538cc2b42eb8cfe2937f57c834b (table test-table [id=44ffdab725034aa18743e4c0b86c3c7d]) successfully deleted
W20260621 05:18:30.885035 14681 connection.cc:535] client connection to 127.0.98.1:35049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260621 05:18:30.885131 14681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:18:30.885404 15163 ts_tablet_manager.cc:1905] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:30.885504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 14887
I20260621 05:18:30.886720 15163 ts_tablet_manager.cc:1918] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:30.886770 15163 log.cc:1196] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/c878f538cc2b42eb8cfe2937f57c834b
I20260621 05:18:30.886981 15163 ts_tablet_manager.cc:1939] T c878f538cc2b42eb8cfe2937f57c834b P ac57c882a8ed47d28eb931b8dcaa3936: Deleting consensus metadata
I20260621 05:18:30.887528 14681 catalog_manager.cc:4841] TS ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967): tablet c878f538cc2b42eb8cfe2937f57c834b (table test-table [id=44ffdab725034aa18743e4c0b86c3c7d]) successfully deleted
I20260621 05:18:30.887650 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:30.887490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.0.1:34364)
0621 05:18:30.887553 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:30.887597 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 05:18:30.887763 14681 catalog_manager.cc:4584] TS 71198d3948694a8a97bed7e98ac29189 (127.0.98.1:35049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c878f538cc2b42eb8cfe2937f57c834b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35049: connect: Connection refused (error 111)
I20260621 05:18:30.890717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 15024
I20260621 05:18:30.895157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 14662
I20260621 05:18:30.900084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39703
--webserver_interface=127.0.98.62
--webserver_port=39705
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:30.976974 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:30.977183 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:30.977202 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:30.978224 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:30.978256 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:30.978271 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:30.978282 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:30.978293 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:30.979606 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:30.979811 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:30.979960 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:30.984416 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:30.984416 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:30.984441 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:30.984642 15185 server_base.cc:1034] running on GCE node
I20260621 05:18:30.984853 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:30.985126 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:30.986263 15185 hybrid_clock.cc:648] HybridClock initialized: now 1782019110986252 us; error 28 us; skew 500 ppm
Jun 21 05:18:30 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:30.988569 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:18:30.988948 15185 webserver.cc:460] Webserver started at http://127.0.98.62:39705/ using document root <none> and password file <none>
I20260621 05:18:30.989171 15185 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:30.989216 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:30.990402 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:30.990990 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:30.991142 15185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:30.991235 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "a2bdc27fffeb4ea1a49e5eb73681356e"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "3fef888d309d1d76b2c7f782bf407f81"
server_key_iv: "7291ab587cbc114ec03e04d71f21c3b3"
server_key_version: "encryptionkey@0"
I20260621 05:18:30.991533 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:31.005699 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:31.008074 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:31.008231 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:31.011900 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39703
I20260621 05:18:31.011929 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39703 every 8 connection(s)
I20260621 05:18:31.012281 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:18:31.013736 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:18:31.014374 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Bootstrap starting.
I20260621 05:18:31.014384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 15185
I20260621 05:18:31.014623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35049
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39703
--webserver_port=33233
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:31.022858 15253 log.cc:824] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:31.024963 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:18:31.025243 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e: Bootstrap complete.
I20260621 05:18:31.027328 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:31.027558 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bdc27fffeb4ea1a49e5eb73681356e, State: Initialized, Role: FOLLOWER
I20260621 05:18:31.027774 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:31.027873 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:18:31.027938 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:18:31.027984 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 2 FOLLOWER]: Advancing to term 3
I20260621 05:18:31.028873 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:31.028991 15253 leader_election.cc:302] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2bdc27fffeb4ea1a49e5eb73681356e; no voters:
I20260621 05:18:31.029191 15253 leader_election.cc:288] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 05:18:31.029222 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 3 FOLLOWER]: Leader election won for term 3
I20260621 05:18:31.029356 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [term 3 LEADER]: Becoming Leader. State: Replica: a2bdc27fffeb4ea1a49e5eb73681356e, State: Running, Role: LEADER
I20260621 05:18:31.029456 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:31.029443 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260621 05:18:31.029743 15257 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2bdc27fffeb4ea1a49e5eb73681356e. Latest consensus state: current_term: 3 leader_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } } }
I20260621 05:18:31.029808 15257 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:31.029968 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bdc27fffeb4ea1a49e5eb73681356e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } } }
I20260621 05:18:31.030038 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bdc27fffeb4ea1a49e5eb73681356e [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:31.030143 15261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:18:31.030901 15261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:18:31.030995 15261 catalog_manager.cc:1259] Loaded cluster ID: c33d6ec7334042d29abef11b347ccffb
I20260621 05:18:31.031028 15261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 05:18:31.031312 15261 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd95ece09ad google::LogMessage::Fail() at ??:0
@ 0x7fd95ece388a google::LogMessage::SendToLog() at ??:0
@ 0x7fd95ece04de google::LogMessage::Flush() at ??:0
@ 0x7fd95ece1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd966c010f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd966c0a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd960bae0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd960ba3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd961a076db start_thread at ??:0
@ 0x7fd95d7de61f clone at ??:0
W20260621 05:18:31.105374 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:31.105576 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:31.105602 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:31.106784 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:31.106827 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:31.106853 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:18:31.108374 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33233
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:31.108583 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:31.108819 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:31.113638 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:31.113636 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:31.113638 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:31.113903 15255 server_base.cc:1034] running on GCE node
I20260621 05:18:31.114079 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:31.114260 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:31.115448 15255 hybrid_clock.cc:648] HybridClock initialized: now 1782019111115437 us; error 28 us; skew 500 ppm
Jun 21 05:18:31 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:31.118616 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:18:31.119123 15255 webserver.cc:460] Webserver started at http://127.0.98.1:33233/ using document root <none> and password file <none>
I20260621 05:18:31.119382 15255 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:31.119459 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:31.121081 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:31.121835 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:31.122030 15255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:31.122135 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "71198d3948694a8a97bed7e98ac29189"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "bde24245807fd77a3e49cceb1b9d657b"
server_key_iv: "4453ccc6117bd0a3b62e2e87397c120b"
server_key_version: "encryptionkey@0"
I20260621 05:18:31.122550 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:31.143764 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:31.146313 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:31.146500 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:31.146791 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:31.147313 15306 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:18:31.148814 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:18:31.148873 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:18:31.148907 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:18:31.149811 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:18:31.149855 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:31.149925 15306 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Bootstrap starting.
I20260621 05:18:31.156972 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35049
I20260621 05:18:31.157282 15413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35049 every 8 connection(s)
I20260621 05:18:31.157477 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:18:31.160015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 15255
I20260621 05:18:31.160315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34943
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39703
--webserver_port=36183
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:31.160355 15306 log.cc:824] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:31.163224 15306 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:18:31.163551 15306 tablet_bootstrap.cc:492] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Bootstrap complete.
I20260621 05:18:31.164776 15306 ts_tablet_manager.cc:1397] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260621 05:18:31.166170 15306 raft_consensus.cc:348] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:31.166513 15306 raft_consensus.cc:729] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71198d3948694a8a97bed7e98ac29189, State: Initialized, Role: FOLLOWER
I20260621 05:18:31.166654 15306 consensus_queue.cc:260] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:31.166925 15306 ts_tablet_manager.cc:1428] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260621 05:18:31.242703 15416 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:18:31.158222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:45327)
0621 05:18:31.158453 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.158504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:31.159121 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:31.242519 (+ 83398us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":108,"threads_started":1}
W20260621 05:18:31.242970 15414 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39703 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39703: Client connection negotiation failed: client connection to 127.0.98.62:39703: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:18:31.243335 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:31.243227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:55183)
0621 05:18:31.243279 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.243318 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 05:18:31.243692 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:31.243575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:59855)
0621 05:18:31.243616 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.243674 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:18:31.243818 15414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:18:31.254467 15418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:31.254667 15418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:31.254693 15418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:31.255851 15418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:31.255908 15418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:31.255951 15418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:18:31.257292 15418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36183
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:31.257472 15418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:31.257642 15418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:31.262669 15424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:31.262656 15425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:31.262710 15418 server_base.cc:1034] running on GCE node
W20260621 05:18:31.262662 15427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:31.263089 15418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:31.263343 15418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:31.264523 15418 hybrid_clock.cc:648] HybridClock initialized: now 1782019111264514 us; error 26 us; skew 500 ppm
Jun 21 05:18:31 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:31.266762 15418 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:18:31.267207 15418 webserver.cc:460] Webserver started at http://127.0.98.2:36183/ using document root <none> and password file <none>
I20260621 05:18:31.267433 15418 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:31.267496 15418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:31.268738 15418 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:31.269255 15434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:31.269414 15418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:31.269511 15418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "c241b29957d04ce0bca52ca5acb68487"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "7a270fe2e5dbc3bff83e028840c76203"
server_key_iv: "ddc07c8d18e87702eaebc3825a4f51c0"
server_key_version: "encryptionkey@0"
I20260621 05:18:31.269860 15418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:31.285707 15418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:31.287505 15418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:31.287719 15418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:31.287940 15418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:31.288233 15418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:31.288283 15418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:31.288313 15418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:31.288342 15418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:31.293814 15418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34943
I20260621 05:18:31.293887 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34943 every 8 connection(s)
I20260621 05:18:31.294226 15418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:18:31.295248 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:31.294893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:43801)
0621 05:18:31.295049 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.295114 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260621 05:18:31.295433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 15418
W20260621 05:18:31.295696 15548 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39703 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39703: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
I20260621 05:18:31.295723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36967
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39703
--webserver_port=36311
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:18:31.296101 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:31.296012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:38291)
0621 05:18:31.296061 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.296085 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 05:18:31.296366 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:31.296295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:38753)
0621 05:18:31.296333 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.296354 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 05:18:31.296479 15548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:18:31.373407 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:31.373651 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:31.373680 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:31.374661 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:31.374693 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:31.374715 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:18:31.376098 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36311
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:31.376288 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:31.376451 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:31.381119 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:31.381187 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:31.381304 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:31.381330 15551 server_base.cc:1034] running on GCE node
I20260621 05:18:31.381525 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:31.381809 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:31.382952 15551 hybrid_clock.cc:648] HybridClock initialized: now 1782019111382939 us; error 32 us; skew 500 ppm
Jun 21 05:18:31 dist-test-slave-czm5 krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:31.385315 15551 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:18:31.385757 15551 webserver.cc:460] Webserver started at http://127.0.98.3:36311/ using document root <none> and password file <none>
I20260621 05:18:31.385931 15551 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:31.385970 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:31.387141 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:18:31.387763 15566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:31.388034 15551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:18:31.388131 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "ac57c882a8ed47d28eb931b8dcaa3936"
format_stamp: "Formatted at 2026-06-21 05:18:29 on dist-test-slave-czm5"
server_key: "ed2cbf54ff384d6c97460f2029037ccf"
server_key_iv: "ca9b16360103824e14115b98c793dec6"
server_key_version: "encryptionkey@0"
I20260621 05:18:31.388458 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:31.406709 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:31.408980 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:31.409150 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:31.409377 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:31.409675 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:31.409729 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:31.409767 15551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:31.409794 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:31.415550 15551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36967
I20260621 05:18:31.415604 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36967 every 8 connection(s)
I20260621 05:18:31.415964 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:18:31.416942 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:31.416606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:49497)
0621 05:18:31.416745 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.416811 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20260621 05:18:31.417382 15680 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39703 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39703: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
I20260621 05:18:31.417665 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:31.417571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:34737)
0621 05:18:31.417617 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.417645 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 05:18:31.417930 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:31.417855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:39309)
0621 05:18:31.417895 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:31.417917 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:18:31.418044 15680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:18:31.420569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 15551
I20260621 05:18:32.244346 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:32.244123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:34231)
0621 05:18:32.244230 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:32.244260 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260621 05:18:32.296945 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:32.296775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:33941)
0621 05:18:32.296871 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:32.296901 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
I20260621 05:18:32.418679 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:32.418474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:52241)
0621 05:18:32.418591 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:32.418628 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260621 05:18:32.517413 15687 raft_consensus.cc:482] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:32.517529 15687 raft_consensus.cc:504] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:32.517925 15687 leader_election.cc:288] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
Jun 21 05:18:32 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 21 05:18:32 dist-test-slave-czm5 krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260621 05:18:32.531317 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:32.518258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39813 (local address 127.0.98.3:36967)
0621 05:18:32.518385 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:32.518397 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:32.518429 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:18:32.518831 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:32.518833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:32.518874 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:32.519031 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:32.519765 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:32.520516 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:32.521435 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:32.521854 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:32.521984 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:32.528089 (+ 6105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:32.528092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:32.528092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:32.528104 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:32.529832 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:32.530142 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:32.530144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:32.530145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:32.530232 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:32.530398 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:32.530399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:32.530399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:32.530682 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:32.530765 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:32.531088 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 05:18:32.531211 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260621 05:18:32.531317 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:32.518173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58981 (local address 127.0.98.2:34943)
0621 05:18:32.518307 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:32.518319 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:32.518351 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:18:32.518790 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:32.518792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:32.518830 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:32.518935 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:32.519716 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:32.520486 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:32.521430 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:32.521854 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:32.521984 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:32.527603 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:32.527607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:32.527607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:32.527646 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:32.529628 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:32.530021 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:32.530023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:32.530023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:32.530097 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:32.530364 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:32.530365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:32.530365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:32.530681 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:32.530765 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:32.531088 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 05:18:32.531212 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260621 05:18:32.531311 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:32.518029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34943 (local address 127.0.98.1:58981)
0621 05:18:32.518066 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:32.518077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:32.518149 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:32.519016 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:32.519018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:32.519389 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:32.519573 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:32.519578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:32.520577 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:32.520578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:32.521343 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:32.521345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:32.521947 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:32.521948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:32.522067 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:32.525877 (+ 3810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:32.525881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:32.527514 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:32.529701 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:32.529706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:32.529708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:32.529941 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:32.530186 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:32.530187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:32.530187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:32.530301 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:32.530757 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:32.530764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:32.530910 (+ 146us) client_negotiation.cc:769] Sending connection context
0621 05:18:32.531022 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 05:18:32.531206 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":31}
I20260621 05:18:32.531375 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:32.518064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.98.1:39813)
0621 05:18:32.518229 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:32.518239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:32.518292 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:32.519029 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:32.519030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:32.519389 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:32.519573 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:32.519578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:32.520625 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:32.520626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:32.521343 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:32.521346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:32.521969 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:32.521970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:32.522070 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:32.525866 (+ 3796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:32.525876 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:32.527974 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:32.529900 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:32.529900 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:32.529901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:32.530078 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:32.530288 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:32.530288 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:32.530289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:32.530341 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:32.530757 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:32.530764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:32.530910 (+ 146us) client_negotiation.cc:769] Sending connection context
0621 05:18:32.531023 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 05:18:32.531257 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":177,"thread_start_us":101,"threads_started":1}
W20260621 05:18:32.532362 15302 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
W20260621 05:18:32.532542 15301 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
I20260621 05:18:32.532589 15301 leader_election.cc:302] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71198d3948694a8a97bed7e98ac29189; no voters: ac57c882a8ed47d28eb931b8dcaa3936, c241b29957d04ce0bca52ca5acb68487
I20260621 05:18:32.532681 15687 raft_consensus.cc:2726] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:18:33.245014 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:33.244818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:40309)
0621 05:18:33.244936 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:33.244968 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260621 05:18:33.297490 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:33.297332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:48749)
0621 05:18:33.297431 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:33.297459 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260621 05:18:33.419441 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:33.419226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:34093)
0621 05:18:33.419370 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:33.419407 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260621 05:18:34.111354 15694 raft_consensus.cc:482] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:34.111449 15694 raft_consensus.cc:504] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:34.111670 15694 leader_election.cc:288] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
W20260621 05:18:34.112021 15301 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
W20260621 05:18:34.112103 15302 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
I20260621 05:18:34.112210 15302 leader_election.cc:302] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71198d3948694a8a97bed7e98ac29189; no voters: ac57c882a8ed47d28eb931b8dcaa3936, c241b29957d04ce0bca52ca5acb68487
I20260621 05:18:34.112298 15694 raft_consensus.cc:2726] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:18:34.245697 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:34.245437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:36705)
0621 05:18:34.245608 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:34.245637 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260621 05:18:34.298059 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:34.297889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:41043)
0621 05:18:34.297994 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:34.298024 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260621 05:18:34.420212 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:34.419987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:42951)
0621 05:18:34.420125 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:34.420174 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260621 05:18:35.246527 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:35.246302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:38561)
0621 05:18:35.246446 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:35.246481 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260621 05:18:35.298905 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:35.298695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:33109)
0621 05:18:35.298839 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:35.298872 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260621 05:18:35.420768 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:35.420577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:58845)
0621 05:18:35.420703 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:35.420735 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260621 05:18:36.247233 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:36.247014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:51029)
0621 05:18:36.247152 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:36.247188 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260621 05:18:36.299520 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:36.299329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:60151)
0621 05:18:36.299452 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:36.299485 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260621 05:18:36.421414 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:36.421224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:36861)
0621 05:18:36.421338 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:36.421376 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260621 05:18:36.448312 15704 raft_consensus.cc:482] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:36.448413 15704 raft_consensus.cc:504] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:36.448621 15704 leader_election.cc:288] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
W20260621 05:18:36.448968 15301 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
W20260621 05:18:36.449045 15302 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
I20260621 05:18:36.449131 15302 leader_election.cc:302] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71198d3948694a8a97bed7e98ac29189; no voters: ac57c882a8ed47d28eb931b8dcaa3936, c241b29957d04ce0bca52ca5acb68487
I20260621 05:18:36.449214 15704 raft_consensus.cc:2726] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:18:37.247977 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:37.247769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:47539)
0621 05:18:37.247896 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:37.247931 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260621 05:18:37.300112 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:37.299942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:45785)
0621 05:18:37.300047 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:37.300079 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":51,"threads_started":1}
I20260621 05:18:37.421952 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:37.421773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:55013)
0621 05:18:37.421896 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:37.421921 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260621 05:18:38.248615 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:38.248419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:42821)
0621 05:18:38.248535 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:38.248568 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260621 05:18:38.300674 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:38.300502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:40547)
0621 05:18:38.300611 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:38.300639 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260621 05:18:38.422552 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:38.422324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:56073)
0621 05:18:38.422469 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:38.422507 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260621 05:18:38.602774 15711 raft_consensus.cc:482] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:38.602859 15711 raft_consensus.cc:504] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:38.603108 15711 leader_election.cc:288] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
W20260621 05:18:38.603466 15302 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
W20260621 05:18:38.603542 15301 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
I20260621 05:18:38.603654 15301 leader_election.cc:302] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71198d3948694a8a97bed7e98ac29189; no voters: ac57c882a8ed47d28eb931b8dcaa3936, c241b29957d04ce0bca52ca5acb68487
I20260621 05:18:38.603756 15711 raft_consensus.cc:2726] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:18:39.249548 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:39.249341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:36027)
0621 05:18:39.249464 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:39.249503 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260621 05:18:39.301344 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:39.301125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:57011)
0621 05:18:39.301262 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:39.301294 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260621 05:18:39.423147 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:39.422940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:57007)
0621 05:18:39.423073 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:39.423108 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260621 05:18:40.250180 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:40.249975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:33037)
0621 05:18:40.250102 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:40.250136 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260621 05:18:40.301928 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:40.301726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:42763)
0621 05:18:40.301844 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:40.301875 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":51,"threads_started":1}
I20260621 05:18:40.423817 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:40.423601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:47087)
0621 05:18:40.423731 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:40.423781 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260621 05:18:41.250919 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:41.250718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:55813)
0621 05:18:41.250840 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:41.250874 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260621 05:18:41.302676 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:41.302464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:49513)
0621 05:18:41.302583 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:41.302619 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
I20260621 05:18:41.424372 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:41.424189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:37181)
0621 05:18:41.424297 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:41.424331 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260621 05:18:42.251660 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:42.251438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:39483)
0621 05:18:42.251551 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:42.251580 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260621 05:18:42.303328 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:42.303111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:55099)
0621 05:18:42.303252 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:42.303288 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
I20260621 05:18:42.425017 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:42.424819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:38137)
0621 05:18:42.424939 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:42.424971 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260621 05:18:43.252362 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:43.252135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:36429)
0621 05:18:43.252282 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:43.252315 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":56,"threads_started":1}
I20260621 05:18:43.303953 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:43.303781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:47883)
0621 05:18:43.303886 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:43.303917 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260621 05:18:43.425786 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:43.425593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:53277)
0621 05:18:43.425720 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:43.425750 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260621 05:18:44.253033 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:44.252807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:56065)
0621 05:18:44.252956 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:44.252988 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260621 05:18:44.304524 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:44.304361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:37897)
0621 05:18:44.304464 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:44.304491 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260621 05:18:44.426402 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:44.426173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:47901)
0621 05:18:44.426311 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:44.426352 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260621 05:18:44.895833 15730 raft_consensus.cc:482] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:44.895923 15730 raft_consensus.cc:504] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c241b29957d04ce0bca52ca5acb68487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34943 } } peers { permanent_uuid: "71198d3948694a8a97bed7e98ac29189" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ac57c882a8ed47d28eb931b8dcaa3936" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260621 05:18:44.896173 15730 leader_election.cc:288] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943), ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967)
W20260621 05:18:44.896557 15302 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac57c882a8ed47d28eb931b8dcaa3936 (127.0.98.3:36967): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
W20260621 05:18:44.896605 15301 leader_election.cc:341] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c241b29957d04ce0bca52ca5acb68487 (127.0.98.2:34943): Not found: Tablet not found: c878f538cc2b42eb8cfe2937f57c834b
I20260621 05:18:44.896632 15302 leader_election.cc:302] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71198d3948694a8a97bed7e98ac29189; no voters: ac57c882a8ed47d28eb931b8dcaa3936, c241b29957d04ce0bca52ca5acb68487
I20260621 05:18:44.896703 15730 raft_consensus.cc:2726] T c878f538cc2b42eb8cfe2937f57c834b P 71198d3948694a8a97bed7e98ac29189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:18:45.253796 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:45.253582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:53479)
0621 05:18:45.253716 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:45.253750 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260621 05:18:45.305133 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:45.304935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:60651)
0621 05:18:45.305061 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:45.305095 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260621 05:18:45.426990 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:45.426780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:34987)
0621 05:18:45.426905 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:45.426937 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260621 05:18:46.254477 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:46.254284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:52111)
0621 05:18:46.254406 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:46.254432 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260621 05:18:46.305729 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:46.305562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:42207)
0621 05:18:46.305662 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:46.305690 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
I20260621 05:18:46.427734 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:46.427489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:45773)
0621 05:18:46.427601 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:46.427682 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260621 05:18:46.430433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 15255
I20260621 05:18:46.435724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 15418
I20260621 05:18:46.440007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 15551
2026-06-21T05:18:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18351 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:18:46 dist-test-slave-czm5 krb5kdc[15739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:18:46 dist-test-slave-czm5 krb5kdc[15739](info): set up 2 sockets
Jun 21 05:18:46 dist-test-slave-czm5 krb5kdc[15739](info): commencing operation
krb5kdc: starting...
W20260621 05:18:47.447711 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:18:47 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:18:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:18:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:18:47.558605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40959
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:47.636404 15752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:47.636593 15752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:47.636615 15752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:47.637671 15752 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:18:47.637705 15752 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:18:47.637718 15752 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:18:47.637729 15752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:18:47.637742 15752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:47.637754 15752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:47.637766 15752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:18:47.637778 15752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:18:47.639132 15752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:47.639338 15752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:47.639508 15752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:47.644114 15760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:47.644177 15752 server_base.cc:1034] running on GCE node
W20260621 05:18:47.644119 15757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:47.644122 15758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:47.644593 15752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:47.644901 15752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:47.646066 15752 hybrid_clock.cc:648] HybridClock initialized: now 1782019127646052 us; error 26 us; skew 500 ppm
Jun 21 05:18:47 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:47.648453 15752 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:18:47.648864 15752 webserver.cc:460] Webserver started at http://127.0.98.62:37043/ using document root <none> and password file <none>
I20260621 05:18:47.649092 15752 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:47.649160 15752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:47.649312 15752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:47.650309 15752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "48f88a596aaf4183b48f1e0185cecf91"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "8d59efdfcc52e76dff0a62ebfff9808b"
server_key_iv: "15130cf68f9447c95d7289f78a663ea7"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.650655 15752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "48f88a596aaf4183b48f1e0185cecf91"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "8d59efdfcc52e76dff0a62ebfff9808b"
server_key_iv: "15130cf68f9447c95d7289f78a663ea7"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.651929 15752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:18:47.652626 15767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:47.652796 15752 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:18:47.652894 15752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "48f88a596aaf4183b48f1e0185cecf91"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "8d59efdfcc52e76dff0a62ebfff9808b"
server_key_iv: "15130cf68f9447c95d7289f78a663ea7"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.652963 15752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:47.665771 15752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:47.667429 15752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:47.667579 15752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:47.671311 15752 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40959
I20260621 05:18:47.671339 15819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40959 every 8 connection(s)
I20260621 05:18:47.671717 15752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:18:47.672550 15820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:47.672701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 15752
I20260621 05:18:47.672811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:18:47.673012 392 external_mini_cluster.cc:1428] Setting key a773c5f5e678cd47d52048c1d5d3aaa1
I20260621 05:18:47.675906 15820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Bootstrap starting.
I20260621 05:18:47.676611 15820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:47.676900 15820 log.cc:824] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:47.677587 15820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: No bootstrap required, opened a new log
I20260621 05:18:47.679428 15820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:18:47.679637 15820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:47.679674 15820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f88a596aaf4183b48f1e0185cecf91, State: Initialized, Role: FOLLOWER
Jun 21 05:18:47 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:47.679828 15820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:18:47.679913 15820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:18:47.679932 15820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:18:47.679996 15820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:47.680490 15820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:18:47.680605 15820 leader_election.cc:302] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f88a596aaf4183b48f1e0185cecf91; no voters:
I20260621 05:18:47.680796 15820 leader_election.cc:288] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:18:47.680822 15825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:47.680931 15825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 1 LEADER]: Becoming Leader. State: Replica: 48f88a596aaf4183b48f1e0185cecf91, State: Running, Role: LEADER
I20260621 05:18:47.681052 15820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:18:47.681067 15825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:18:47.681368 15826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f88a596aaf4183b48f1e0185cecf91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } } }
I20260621 05:18:47.681517 15826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:47.682040 15832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:18:47.682547 15832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:18:47.681368 15827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f88a596aaf4183b48f1e0185cecf91. Latest consensus state: current_term: 1 leader_uuid: "48f88a596aaf4183b48f1e0185cecf91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } } }
I20260621 05:18:47.682829 15827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: This master's current role is: LEADER
I20260621 05:18:47.684165 15832 catalog_manager.cc:1347] Generated new cluster ID: bcadd3aff772440785db0d06127c9a0b
I20260621 05:18:47.684219 15832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:18:47.685675 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:47.673537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.0.98.62:40959)
0621 05:18:47.673717 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:47.673726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:47.673756 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:18:47.674174 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:47.674180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:47.674225 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:47.674337 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:47.675134 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:47.676128 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:47.677198 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:47.677697 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:47.677811 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:47.680246 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:47.680250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:47.680250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:47.680260 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:47.684023 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:47.684463 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:47.684466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:47.684466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:47.684566 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:47.684787 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:47.684787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:47.684788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:47.685046 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:47.685201 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:47.685372 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 05:18:47.685488 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260621 05:18:47.704542 15832 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:18:47.705144 15832 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:18:47.718398 15832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Generated new TSK 0
I20260621 05:18:47.718592 15832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:18:47.746356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40959
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:47.825850 15846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:47.826046 15846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:47.826066 15846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:47.827157 15846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:47.827191 15846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:47.827215 15846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:18:47.828660 15846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:47.828850 15846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:47.829028 15846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:47.834009 15851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:47.833983 15852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:47.834002 15846 server_base.cc:1034] running on GCE node
W20260621 05:18:47.834036 15854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:47.834412 15846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:47.834697 15846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:47.835845 15846 hybrid_clock.cc:648] HybridClock initialized: now 1782019127835833 us; error 29 us; skew 500 ppm
Jun 21 05:18:47 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:47.838165 15846 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:18:47.838573 15846 webserver.cc:460] Webserver started at http://127.0.98.1:34757/ using document root <none> and password file <none>
I20260621 05:18:47.838806 15846 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:47.838871 15846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:47.839015 15846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:47.840055 15846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "69587104581d4b80a1e970d6cd9ac179"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "be877c3aa72f25f3e5e334adca602b40"
server_key_iv: "8ac836bd2b1b1a64c0bc2de0d1323813"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.840400 15846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "69587104581d4b80a1e970d6cd9ac179"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "be877c3aa72f25f3e5e334adca602b40"
server_key_iv: "8ac836bd2b1b1a64c0bc2de0d1323813"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.841624 15846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:18:47.842309 15861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:47.842476 15846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:47.842568 15846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "69587104581d4b80a1e970d6cd9ac179"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "be877c3aa72f25f3e5e334adca602b40"
server_key_iv: "8ac836bd2b1b1a64c0bc2de0d1323813"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.842644 15846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:47.850309 15846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:47.852047 15846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:47.852205 15846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:47.852423 15846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:47.852713 15846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:47.852766 15846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:47.852803 15846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:47.852831 15846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:47.858439 15846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38499
I20260621 05:18:47.858481 15974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38499 every 8 connection(s)
I20260621 05:18:47.858808 15846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:18:47.860926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 15846
I20260621 05:18:47.861040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:18:47.861244 392 external_mini_cluster.cc:1428] Setting key 94ad56108d050fd9cfc91e87e04a016a
Jun 21 05:18:47 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:47.868512 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:47.859592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34489 (local address 127.0.98.62:40959)
0621 05:18:47.859658 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:47.859660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:47.859849 (+ 189us) server_negotiation.cc:407] Connection header received
0621 05:18:47.860351 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:47.860352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:47.860364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:47.860417 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:47.861575 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:47.862075 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:47.863118 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:47.863334 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:47.863398 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:47.865494 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:47.865495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:47.865496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:47.865505 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:47.867089 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:47.867499 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:47.867500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:47.867500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:47.867582 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:47.867872 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:47.867873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:47.867874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:47.868020 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:47.868096 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:47.868319 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 05:18:47.868408 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:18:47.868527 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:47.859564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:34489)
0621 05:18:47.859737 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:47.859772 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:47.860237 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:47.860485 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:47.860488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:47.861216 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:47.861478 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:47.861484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:47.862168 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:47.862169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:47.863025 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:47.863028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:47.863424 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:47.863425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:47.863569 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:47.864060 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:47.864067 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:47.865404 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:47.867213 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:47.867218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:47.867220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:47.867429 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:47.867706 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:47.867707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:47.867707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:47.867803 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:47.868088 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:47.868092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:47.868195 (+ 103us) client_negotiation.cc:769] Sending connection context
0621 05:18:47.868273 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:18:47.868382 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260621 05:18:47.869042 15975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40959
I20260621 05:18:47.869143 15975 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:47.869323 15975 heartbeater.cc:511] Master 127.0.98.62:40959 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260621 05:18:47.869867 15784 ts_manager.cc:194] Registered new tserver with Master: 69587104581d4b80a1e970d6cd9ac179 (127.0.98.1:38499)
I20260621 05:18:47.870460 15784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:18:47.886330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40959
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:47.964792 15980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:47.964977 15980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:47.964998 15980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:47.966089 15980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:47.966126 15980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:47.966181 15980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:18:47.967526 15980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:47.967747 15980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:47.967954 15980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:47.972648 15988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:47.972648 15986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:47.972759 15985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:47.972848 15980 server_base.cc:1034] running on GCE node
I20260621 05:18:47.973071 15980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:47.973309 15980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:47.974458 15980 hybrid_clock.cc:648] HybridClock initialized: now 1782019127974428 us; error 33 us; skew 500 ppm
Jun 21 05:18:47 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:47.976794 15980 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:18:47.977236 15980 webserver.cc:460] Webserver started at http://127.0.98.2:35337/ using document root <none> and password file <none>
I20260621 05:18:47.977468 15980 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:47.977530 15980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:47.977679 15980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:47.978624 15980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "dfbd3850162e4365a881511587fcf04d"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "c1d0687a0d6ca5a2cca5e6c4e717e2b6"
server_key_iv: "694f2cac3f9672b2d967549f01636794"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.978981 15980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dfbd3850162e4365a881511587fcf04d"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "c1d0687a0d6ca5a2cca5e6c4e717e2b6"
server_key_iv: "694f2cac3f9672b2d967549f01636794"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.980289 15980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:18:47.980970 15995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:47.981132 15980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:47.981249 15980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "dfbd3850162e4365a881511587fcf04d"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "c1d0687a0d6ca5a2cca5e6c4e717e2b6"
server_key_iv: "694f2cac3f9672b2d967549f01636794"
server_key_version: "encryptionkey@0"
I20260621 05:18:47.981317 15980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:47.989920 15980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:47.991544 15980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:47.991719 15980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:47.991914 15980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:47.992189 15980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:47.992244 15980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:47.992281 15980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:47.992308 15980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:47.997937 15980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36919
I20260621 05:18:47.997969 16108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36919 every 8 connection(s)
I20260621 05:18:47.998257 15980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:18:48.000658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 15980
I20260621 05:18:48.000768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:18:48.000929 392 external_mini_cluster.cc:1428] Setting key ebfa425027468f88e68fcceecd3dc89c
Jun 21 05:18:48 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:48.007992 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:47.998917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50761 (local address 127.0.98.62:40959)
0621 05:18:47.998979 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:47.998981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:47.999171 (+ 190us) server_negotiation.cc:407] Connection header received
0621 05:18:47.999584 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:47.999585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:47.999593 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:47.999697 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:48.000505 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.001039 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.002280 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.002458 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.002508 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.004967 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:48.004969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:48.004969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:48.004977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:48.006570 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:48.006972 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:48.006973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:48.006974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:48.007035 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:48.007305 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:48.007306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:48.007306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:48.007424 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:48.007510 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.007821 (+ 311us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.007903 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:18:48.008162 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:47.998892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:50761)
0621 05:18:47.999058 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:47.999090 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:47.999512 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:47.999767 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:47.999770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:48.000273 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:48.000433 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.000437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.001146 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.001147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.002201 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.002204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.002535 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.002536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.002676 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.003208 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:48.003216 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:48.004860 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:48.006655 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:48.006661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:48.006663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:48.006880 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:48.007110 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:48.007111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:48.007111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:48.007235 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:48.007502 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:48.007508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:48.007681 (+ 173us) client_negotiation.cc:769] Sending connection context
0621 05:18:48.007797 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 05:18:48.007944 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260621 05:18:48.008682 16109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40959
I20260621 05:18:48.008790 16109 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:48.008970 16109 heartbeater.cc:511] Master 127.0.98.62:40959 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260621 05:18:48.009347 15784 ts_manager.cc:194] Registered new tserver with Master: dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
I20260621 05:18:48.009723 15784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:18:48.025563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40959
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:18:48.106088 16114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:18:48.106319 16114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:18:48.106343 16114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:18:48.107688 16114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:18:48.107728 16114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:18:48.107754 16114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:18:48.109212 16114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:18:48.109472 16114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:18:48.109687 16114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:18:48.114691 16122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:48.114666 16120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:18:48.114651 16119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:18:48.114717 16114 server_base.cc:1034] running on GCE node
I20260621 05:18:48.115059 16114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:18:48.115288 16114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:18:48.116420 16114 hybrid_clock.cc:648] HybridClock initialized: now 1782019128116411 us; error 25 us; skew 500 ppm
Jun 21 05:18:48 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:18:48.118722 16114 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:18:48.119158 16114 webserver.cc:460] Webserver started at http://127.0.98.3:41325/ using document root <none> and password file <none>
I20260621 05:18:48.119400 16114 fs_manager.cc:359] Metadata directory not provided
I20260621 05:18:48.119443 16114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:18:48.119578 16114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:18:48.120564 16114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "d3d3b9444b314c7cb8a00b43836a0d85"
format_stamp: "Formatted at 2026-06-21 05:18:48 on dist-test-slave-czm5"
server_key: "800856dee04937e1269c724f34187c96"
server_key_iv: "4e14e9dd25e12cd347f1b1ee8909cee5"
server_key_version: "encryptionkey@0"
I20260621 05:18:48.120843 16114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d3d3b9444b314c7cb8a00b43836a0d85"
format_stamp: "Formatted at 2026-06-21 05:18:48 on dist-test-slave-czm5"
server_key: "800856dee04937e1269c724f34187c96"
server_key_iv: "4e14e9dd25e12cd347f1b1ee8909cee5"
server_key_version: "encryptionkey@0"
I20260621 05:18:48.121994 16114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:18:48.122588 16129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:48.122767 16114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:18:48.122860 16114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "d3d3b9444b314c7cb8a00b43836a0d85"
format_stamp: "Formatted at 2026-06-21 05:18:48 on dist-test-slave-czm5"
server_key: "800856dee04937e1269c724f34187c96"
server_key_iv: "4e14e9dd25e12cd347f1b1ee8909cee5"
server_key_version: "encryptionkey@0"
I20260621 05:18:48.122947 16114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:18:48.141324 16114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:18:48.143004 16114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:18:48.143163 16114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:18:48.143411 16114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:18:48.143785 16114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:18:48.143841 16114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:48.143880 16114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:18:48.143908 16114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:18:48.149765 16114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39557
I20260621 05:18:48.149801 16242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39557 every 8 connection(s)
I20260621 05:18:48.150122 16114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:18:48 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:18:48.159241 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.150860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44079 (local address 127.0.98.62:40959)
0621 05:18:48.150940 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:48.150942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:48.151112 (+ 170us) server_negotiation.cc:407] Connection header received
0621 05:18:48.151645 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:48.151646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:48.151659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:48.151715 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:48.152819 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.153330 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.154310 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.154529 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.154616 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.156745 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:48.156747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:48.156747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:48.156755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:48.157930 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:48.158324 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:48.158326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:48.158326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:48.158389 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:48.158617 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:48.158618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:48.158618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:48.158735 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:48.158812 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.159066 (+ 254us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.159145 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:18:48.159402 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.150831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:44079)
0621 05:18:48.151002 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:48.151034 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:48.151522 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:48.151792 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:48.151796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:48.152511 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:18:48.152717 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.152722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.153424 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.153425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.154217 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.154220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.154624 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.154625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.154759 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.155297 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:18:48.155305 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:18:48.156646 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:18:48.158024 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:48.158031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:48.158033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:48.158241 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:48.158452 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:18:48.158453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:18:48.158453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:18:48.158552 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:18:48.158802 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:18:48.158808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:18:48.158939 (+ 131us) client_negotiation.cc:769] Sending connection context
0621 05:18:48.159035 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:18:48.159181 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260621 05:18:48.159904 16243 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40959
I20260621 05:18:48.159977 16243 heartbeater.cc:464] Registering TS with master...
I20260621 05:18:48.160180 16243 heartbeater.cc:511] Master 127.0.98.62:40959 requested a full tablet report, sending...
I20260621 05:18:48.160181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 16114
I20260621 05:18:48.160269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:18:48.160481 392 external_mini_cluster.cc:1428] Setting key aa227cf4ca631dcb0cb658651e3256bc
I20260621 05:18:48.160632 15784 ts_manager.cc:194] Registered new tserver with Master: d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557)
I20260621 05:18:48.161109 15784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44079
I20260621 05:18:48.161358 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:18:48.168327 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.162741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.0.98.62:40959)
0621 05:18:48.162797 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:48.162799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:48.162878 (+ 79us) server_negotiation.cc:407] Connection header received
0621 05:18:48.162923 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:48.162924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:48.162932 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:48.162974 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:18:48.163501 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.164020 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.164712 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.164884 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.164932 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.166078 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:18:48.166078 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:18:48.166079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:18:48.166083 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:18:48.167187 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:48.167479 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:48.167480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:48.167480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:48.167529 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:18:48.167772 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:18:48.167774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:18:48.167774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:18:48.167917 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:18:48.168039 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.168159 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.168239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:18:48.169842 15784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:18:48.171360 15784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:18:48.177723 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.173564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.0.98.2:36919)
0621 05:18:48.173731 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:48.173737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:48.173912 (+ 175us) server_negotiation.cc:407] Connection header received
0621 05:18:48.174025 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:48.174027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:48.174044 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:48.174160 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:48.174295 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.175421 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.176807 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.177315 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.177495 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.177564 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.177590 (+ 26us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.177628 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260621 05:18:48.177742 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.173471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36919 (local address 127.0.0.1:41160)
0621 05:18:48.173796 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:48.173837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:48.173917 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:48.174160 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:48.174163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:48.174165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:48.174241 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.174243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.175528 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.175529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.176719 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.176721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.177405 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.177406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.177490 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.177505 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 05:18:48.177564 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 05:18:48.177617 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":59,"thread_start_us":56,"threads_started":1}
I20260621 05:18:48.177913 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.173908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.0.98.1:38499)
0621 05:18:48.174080 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:48.174084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:48.174094 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:18:48.174131 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:48.174133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:48.174148 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:48.174449 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:48.174519 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.175176 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.176792 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.177471 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.177532 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.177586 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.177793 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.177836 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260621 05:18:48.177955 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.173507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39557 (local address 127.0.0.1:59524)
0621 05:18:48.173877 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:48.173887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:48.173933 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:48.174289 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:48.174290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:48.174292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:48.174343 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.174345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.175562 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.175563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.176572 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.176574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.177504 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.177505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.177574 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.177578 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:18:48.177619 (+ 41us) client_negotiation.cc:240] Negotiation successful
0621 05:18:48.177659 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":77,"threads_started":1}
I20260621 05:18:48.177973 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.173586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.0.98.3:39557)
0621 05:18:48.173753 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:48.173756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:48.174072 (+ 316us) server_negotiation.cc:407] Connection header received
0621 05:18:48.174113 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:48.174115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:48.174130 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:48.174197 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:48.174422 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.175461 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.176653 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.177363 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.177434 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.177824 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.177847 (+ 23us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.177886 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260621 05:18:48.178115 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.173670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38499 (local address 127.0.0.1:50966)
0621 05:18:48.174022 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:48.174031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:48.174067 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:48.174338 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:48.174338 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:48.174340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:48.174378 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.174379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.175275 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.175276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.176709 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.176712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.177616 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.177617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.177666 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.177669 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:18:48.177714 (+ 45us) client_negotiation.cc:240] Negotiation successful
0621 05:18:48.177751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260621 05:18:48.178920 16043 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb552e446964177908457277d0895ce (DEFAULT_TABLE table=test-table [id=d54a74803def4bebae06e5d9dff32276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:48.178866 16177 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb552e446964177908457277d0895ce (DEFAULT_TABLE table=test-table [id=d54a74803def4bebae06e5d9dff32276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:48.178929 15909 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb552e446964177908457277d0895ce (DEFAULT_TABLE table=test-table [id=d54a74803def4bebae06e5d9dff32276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:18:48.179257 15909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb552e446964177908457277d0895ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:48.179257 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb552e446964177908457277d0895ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:48.179266 16043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb552e446964177908457277d0895ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:18:48.182057 16257 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Bootstrap starting.
I20260621 05:18:48.182685 16258 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Bootstrap starting.
I20260621 05:18:48.182873 16257 tablet_bootstrap.cc:654] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:48.182901 16259 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: Bootstrap starting.
I20260621 05:18:48.183378 16257 log.cc:824] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:48.183554 16259 tablet_bootstrap.cc:654] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:48.184098 16257 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: No bootstrap required, opened a new log
I20260621 05:18:48.184154 16257 ts_tablet_manager.cc:1397] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:18:48.184217 16258 tablet_bootstrap.cc:654] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Neither blocks nor log segments found. Creating new log.
I20260621 05:18:48.184725 16258 log.cc:824] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:48.185444 16259 log.cc:824] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: Log is configured to *not* fsync() on all Append() calls
I20260621 05:18:48.185829 16257 raft_consensus.cc:348] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.185968 16258 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: No bootstrap required, opened a new log
I20260621 05:18:48.186028 16257 raft_consensus.cc:374] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:48.186046 16258 ts_tablet_manager.cc:1397] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 05:18:48.186092 16257 raft_consensus.cc:729] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbd3850162e4365a881511587fcf04d, State: Initialized, Role: FOLLOWER
I20260621 05:18:48.186192 16259 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: No bootstrap required, opened a new log
I20260621 05:18:48.186252 16259 ts_tablet_manager.cc:1397] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:18:48.186214 16257 consensus_queue.cc:260] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.186496 16257 ts_tablet_manager.cc:1428] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:18:48.186496 16109 heartbeater.cc:503] Master 127.0.98.62:40959 was elected leader, sending a full tablet report...
I20260621 05:18:48.187984 16258 raft_consensus.cc:348] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.188158 16258 raft_consensus.cc:374] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:48.188192 16258 raft_consensus.cc:729] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69587104581d4b80a1e970d6cd9ac179, State: Initialized, Role: FOLLOWER
I20260621 05:18:48.188114 16259 raft_consensus.cc:348] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.188275 16259 raft_consensus.cc:374] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:18:48.188319 16259 raft_consensus.cc:729] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d3b9444b314c7cb8a00b43836a0d85, State: Initialized, Role: FOLLOWER
I20260621 05:18:48.188339 16258 consensus_queue.cc:260] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.188418 16259 consensus_queue.cc:260] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.188643 16258 ts_tablet_manager.cc:1428] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:18:48.188675 16259 ts_tablet_manager.cc:1428] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:18:48.188678 15975 heartbeater.cc:503] Master 127.0.98.62:40959 was elected leader, sending a full tablet report...
I20260621 05:18:48.188746 16243 heartbeater.cc:503] Master 127.0.98.62:40959 was elected leader, sending a full tablet report...
W20260621 05:18:48.248798 16110 tablet.cc:2369] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:18:48.291960 16264 raft_consensus.cc:482] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:18:48.292114 16264 raft_consensus.cc:504] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.292536 16264 leader_election.cc:288] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919), d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557)
I20260621 05:18:48.296383 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.292818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33471 (local address 127.0.98.2:36919)
0621 05:18:48.292860 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:48.292862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:48.292953 (+ 91us) server_negotiation.cc:407] Connection header received
0621 05:18:48.293010 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:48.293011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:48.293022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:48.293073 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:48.293240 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.293921 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.295425 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.295994 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.296098 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.296108 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.296277 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.296314 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:18:48.296382 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.292784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48227 (local address 127.0.98.3:39557)
0621 05:18:48.292828 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:48.292830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:48.292839 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:18:48.292869 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:48.292870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:48.292883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:48.292967 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:18:48.293141 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.293902 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.295425 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.295994 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.296098 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.296108 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.296282 (+ 174us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.296318 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:18:48.296422 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.292679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39557 (local address 127.0.98.1:48227)
0621 05:18:48.292739 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:48.292753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:48.292800 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:48.292972 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:48.292973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:48.292975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:48.293050 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.293052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.293992 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.293993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.295310 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.295313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.296099 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.296100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.296198 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.296201 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:18:48.296253 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:18:48.296312 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":51}
I20260621 05:18:48.296428 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.292691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36919 (local address 127.0.98.1:33471)
0621 05:18:48.292878 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:48.292889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:18:48.292960 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:18:48.293092 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:18:48.293093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:18:48.293094 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:18:48.293162 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.293164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.293992 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.293993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.295311 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:18:48.295313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.296099 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.296100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:18:48.296190 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.296193 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:18:48.296253 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 05:18:48.296312 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":115,"thread_start_us":109,"threads_started":1}
I20260621 05:18:48.296911 16063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb552e446964177908457277d0895ce" candidate_uuid: "69587104581d4b80a1e970d6cd9ac179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbd3850162e4365a881511587fcf04d" is_pre_election: true
I20260621 05:18:48.296943 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb552e446964177908457277d0895ce" candidate_uuid: "69587104581d4b80a1e970d6cd9ac179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" is_pre_election: true
I20260621 05:18:48.297075 16063 raft_consensus.cc:2454] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69587104581d4b80a1e970d6cd9ac179 in term 0.
I20260621 05:18:48.297075 16197 raft_consensus.cc:2454] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69587104581d4b80a1e970d6cd9ac179 in term 0.
I20260621 05:18:48.297289 15864 leader_election.cc:302] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179, d3d3b9444b314c7cb8a00b43836a0d85; no voters:
I20260621 05:18:48.297461 16264 raft_consensus.cc:2781] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:18:48.297499 16264 raft_consensus.cc:482] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:18:48.297514 16264 raft_consensus.cc:3037] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:48.298158 16264 raft_consensus.cc:504] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.298293 16264 leader_election.cc:288] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 1 election: Requested vote from peers dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919), d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557)
I20260621 05:18:48.298436 16063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb552e446964177908457277d0895ce" candidate_uuid: "69587104581d4b80a1e970d6cd9ac179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbd3850162e4365a881511587fcf04d"
I20260621 05:18:48.298448 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb552e446964177908457277d0895ce" candidate_uuid: "69587104581d4b80a1e970d6cd9ac179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d3b9444b314c7cb8a00b43836a0d85"
I20260621 05:18:48.298513 16063 raft_consensus.cc:3037] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:48.298513 16197 raft_consensus.cc:3037] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:18:48.299340 16063 raft_consensus.cc:2454] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69587104581d4b80a1e970d6cd9ac179 in term 1.
I20260621 05:18:48.299340 16197 raft_consensus.cc:2454] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69587104581d4b80a1e970d6cd9ac179 in term 1.
I20260621 05:18:48.299508 15863 leader_election.cc:302] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179, dfbd3850162e4365a881511587fcf04d; no voters:
I20260621 05:18:48.299610 16264 raft_consensus.cc:2781] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:18:48.299794 16264 raft_consensus.cc:686] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 1 LEADER]: Becoming Leader. State: Replica: 69587104581d4b80a1e970d6cd9ac179, State: Running, Role: LEADER
I20260621 05:18:48.299898 16264 consensus_queue.cc:237] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:18:48.300709 15783 catalog_manager.cc:5495] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69587104581d4b80a1e970d6cd9ac179 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "69587104581d4b80a1e970d6cd9ac179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } health_report { overall_health: UNKNOWN } } }
I20260621 05:18:48.324838 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.322717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.0.98.1:38499)
0621 05:18:48.322775 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:18:48.322777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:18:48.322800 (+ 23us) server_negotiation.cc:407] Connection header received
0621 05:18:48.322829 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:18:48.322830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:18:48.322847 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:18:48.322898 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:18:48.323008 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.323471 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.324191 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:18:48.324369 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:18:48.324418 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:18:48.324543 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:18:48.324636 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:18:48.324698 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:18:48.324720 (+ 22us) server_negotiation.cc:299] Negotiation successful
0621 05:18:48.324736 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:18:48.326683 16197 raft_consensus.cc:1261] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 1 FOLLOWER]: Refusing update from remote peer 69587104581d4b80a1e970d6cd9ac179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:48.326767 16063 raft_consensus.cc:1261] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 1 FOLLOWER]: Refusing update from remote peer 69587104581d4b80a1e970d6cd9ac179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:18:48.327009 16264 consensus_queue.cc:1035] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:48.327165 16267 consensus_queue.cc:1035] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:18:48.329962 16271 mvcc.cc:205] Tried to move back new op lower bound from 7299150349623570432 to 7299150349516992512. Current Snapshot: MvccSnapshot[applied={T|T < 7299150349623570432}]
I20260621 05:18:48.330492 16272 mvcc.cc:205] Tried to move back new op lower bound from 7299150349623570432 to 7299150349516992512. Current Snapshot: MvccSnapshot[applied={T|T < 7299150349623570432 or (T in {7299150349623570432})}]
I20260621 05:18:48.332314 15784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:18:48.332422 15784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:18:48.333184 15784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Sending DeleteTablet for 3 replicas of tablet 9cb552e446964177908457277d0895ce
I20260621 05:18:48.333519 16177 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb552e446964177908457277d0895ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59524
I20260621 05:18:48.333520 15909 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb552e446964177908457277d0895ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50966
I20260621 05:18:48.333535 16043 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb552e446964177908457277d0895ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41160
I20260621 05:18:48.333750 16282 tablet_replica.cc:330] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: stopping tablet replica
I20260621 05:18:48.333881 16282 raft_consensus.cc:2229] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:18:48.333884 16281 tablet_replica.cc:330] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: stopping tablet replica
I20260621 05:18:48.333897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 15846
I20260621 05:18:48.334050 16282 raft_consensus.cc:2258] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:48.335064 16282 ts_tablet_manager.cc:1905] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:48.333976 16281 raft_consensus.cc:2229] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:18:48.335544 16281 raft_consensus.cc:2258] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:18:48.335915 16281 ts_tablet_manager.cc:1905] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:18:48.337114 16281 ts_tablet_manager.cc:1918] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:48.337152 16282 ts_tablet_manager.cc:1918] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:18:48.337203 16281 log.cc:1196] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/9cb552e446964177908457277d0895ce
I20260621 05:18:48.337221 16282 log.cc:1196] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/9cb552e446964177908457277d0895ce
I20260621 05:18:48.337484 16282 ts_tablet_manager.cc:1939] T 9cb552e446964177908457277d0895ce P d3d3b9444b314c7cb8a00b43836a0d85: Deleting consensus metadata
I20260621 05:18:48.337491 16281 ts_tablet_manager.cc:1939] T 9cb552e446964177908457277d0895ce P dfbd3850162e4365a881511587fcf04d: Deleting consensus metadata
I20260621 05:18:48.337882 15769 catalog_manager.cc:4841] TS dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919): tablet 9cb552e446964177908457277d0895ce (table test-table [id=d54a74803def4bebae06e5d9dff32276]) successfully deleted
I20260621 05:18:48.337889 15770 catalog_manager.cc:4841] TS d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557): tablet 9cb552e446964177908457277d0895ce (table test-table [id=d54a74803def4bebae06e5d9dff32276]) successfully deleted
W20260621 05:18:48.339473 15769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38499 (error 108)
I20260621 05:18:48.339704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 15980
I20260621 05:18:48.340266 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:18:48.339792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38499 (local address 127.0.0.1:50988)
0621 05:18:48.340200 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0621 05:18:48.340232 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
W20260621 05:18:48.340409 15769 catalog_manager.cc:4584] TS 69587104581d4b80a1e970d6cd9ac179 (127.0.98.1:38499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cb552e446964177908457277d0895ce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38499: connect: Connection refused (error 111)
I20260621 05:18:48.344349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 16114
I20260621 05:18:48.348645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 15752
I20260621 05:19:08.353718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40959
--webserver_interface=127.0.98.62
--webserver_port=37043
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:08.434134 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:08.434327 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:08.434350 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:08.435511 16284 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:19:08.435549 16284 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:19:08.435563 16284 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:19:08.435575 16284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:08.435640 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:08.435659 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:08.435691 16284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:08.435719 16284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:08.437248 16284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:08.437511 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:08.437731 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:08.445807 16293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:08.445799 16290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:08.445803 16291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:08.446034 16284 server_base.cc:1034] running on GCE node
I20260621 05:19:08.446328 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:08.446622 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:08.447786 16284 hybrid_clock.cc:648] HybridClock initialized: now 1782019148447771 us; error 43 us; skew 500 ppm
Jun 21 05:19:08 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:08.450196 16284 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:19:08.450596 16284 webserver.cc:460] Webserver started at http://127.0.98.62:37043/ using document root <none> and password file <none>
I20260621 05:19:08.450816 16284 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:08.450908 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:08.452260 16284 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:19:08.452836 16300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.453008 16284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:08.453110 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "48f88a596aaf4183b48f1e0185cecf91"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "8d59efdfcc52e76dff0a62ebfff9808b"
server_key_iv: "15130cf68f9447c95d7289f78a663ea7"
server_key_version: "encryptionkey@0"
I20260621 05:19:08.453481 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:08.473690 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:08.476048 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:08.476213 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:08.480144 16284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40959
I20260621 05:19:08.480175 16352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40959 every 8 connection(s)
I20260621 05:19:08.480562 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:08.481976 16353 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:19:08.482617 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Bootstrap starting.
I20260621 05:19:08.488997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 16284
I20260621 05:19:08.489326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38499
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40959
--webserver_port=34757
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:19:08.491310 16353 log.cc:824] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:08.493269 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:19:08.493553 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Bootstrap complete.
I20260621 05:19:08.495868 16353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:19:08.496091 16353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f88a596aaf4183b48f1e0185cecf91, State: Initialized, Role: FOLLOWER
I20260621 05:19:08.496268 16353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:19:08.496326 16353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:08.496404 16353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:08.496461 16353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:19:08.497452 16353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:19:08.497534 16353 leader_election.cc:302] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f88a596aaf4183b48f1e0185cecf91; no voters:
I20260621 05:19:08.497720 16353 leader_election.cc:288] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:19:08.497752 16357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:19:08.497864 16357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 2 LEADER]: Becoming Leader. State: Replica: 48f88a596aaf4183b48f1e0185cecf91, State: Running, Role: LEADER
I20260621 05:19:08.497979 16353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:08.498034 16357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:19:08.498389 16359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f88a596aaf4183b48f1e0185cecf91. Latest consensus state: current_term: 2 leader_uuid: "48f88a596aaf4183b48f1e0185cecf91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } } }
I20260621 05:19:08.498399 16358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "48f88a596aaf4183b48f1e0185cecf91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } } }
I20260621 05:19:08.498498 16359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:08.498509 16358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:08.498747 16365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:08.499621 16365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:08.499814 16365 catalog_manager.cc:1259] Loaded cluster ID: bcadd3aff772440785db0d06127c9a0b
I20260621 05:19:08.500005 16365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:08.501104 16365 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:08.501271 16365 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Loaded TSK: 0
I20260621 05:19:08.515139 16365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Generated new TSK 1
I20260621 05:19:08.515244 16365 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:19:08.572418 16355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:08.572618 16355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:08.572640 16355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:08.573769 16355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:08.573834 16355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:08.573892 16355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:19:08.575368 16355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34757
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:08.575659 16355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:08.575906 16355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:08.580796 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:08.580801 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:08.581110 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:08.581164 16355 server_base.cc:1034] running on GCE node
I20260621 05:19:08.581367 16355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:08.581638 16355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:08.582790 16355 hybrid_clock.cc:648] HybridClock initialized: now 1782019148582774 us; error 35 us; skew 500 ppm
Jun 21 05:19:08 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:08.585304 16355 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:19:08.585790 16355 webserver.cc:460] Webserver started at http://127.0.98.1:34757/ using document root <none> and password file <none>
I20260621 05:19:08.586041 16355 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:08.586097 16355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:08.587469 16355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:08.588200 16390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.588383 16355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:19:08.588486 16355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "69587104581d4b80a1e970d6cd9ac179"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "be877c3aa72f25f3e5e334adca602b40"
server_key_iv: "8ac836bd2b1b1a64c0bc2de0d1323813"
server_key_version: "encryptionkey@0"
I20260621 05:19:08.588822 16355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:08.599776 16355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:08.601950 16355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:08.602108 16355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:08.602347 16355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:08.602787 16397 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:19:08.604638 16355 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:19:08.604700 16355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:19:08.604741 16355 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:19:08.605357 16355 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:19:08.605413 16355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:19:08.605486 16397 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Bootstrap starting.
I20260621 05:19:08.612610 16355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38499
I20260621 05:19:08.612995 16355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:19:08.613571 16504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38499 every 8 connection(s)
I20260621 05:19:08.614506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 16355
I20260621 05:19:08.614789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36919
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40959
--webserver_port=35337
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:19:08.616333 16397 log.cc:824] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:08.618741 16397 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:19:08.619025 16397 tablet_bootstrap.cc:492] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Bootstrap complete.
I20260621 05:19:08.620188 16397 ts_tablet_manager.cc:1397] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.013s
I20260621 05:19:08.621420 16397 raft_consensus.cc:348] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:19:08.621735 16397 raft_consensus.cc:729] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69587104581d4b80a1e970d6cd9ac179, State: Initialized, Role: FOLLOWER
Jun 21 05:19:08 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:08.621888 16397 consensus_queue.cc:260] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } }
I20260621 05:19:08.622131 16397 ts_tablet_manager.cc:1428] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:19:08.626772 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.613756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:57929)
0621 05:19:08.615755 (+ 1999us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.615807 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:08.616413 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:08.617065 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:08.617069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:08.617343 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:08.617545 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.617549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.618635 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.618636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.619537 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.619540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.620160 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.620161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.620288 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.620848 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:08.620856 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:08.622365 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:08.624790 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:08.624798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:08.624800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:08.625182 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:08.625499 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:08.625500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:08.625500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:08.625630 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:08.626127 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:08.626133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:08.626280 (+ 147us) client_negotiation.cc:769] Sending connection context
0621 05:19:08.626390 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 05:19:08.626563 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1947,"thread_start_us":82,"threads_started":1}
I20260621 05:19:08.626865 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.613875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57929 (local address 127.0.98.62:40959)
0621 05:19:08.616655 (+ 2780us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.616665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.616704 (+ 39us) server_negotiation.cc:407] Connection header received
0621 05:19:08.616836 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.616842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.616898 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.617038 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:08.617658 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.618518 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.619658 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.620059 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.620186 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.622483 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:08.622488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:08.622488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:08.622499 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:08.624685 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:08.625285 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:08.625289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:08.625289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:08.625411 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:08.625707 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:08.625708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:08.625709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:08.626038 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:08.626134 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.626472 (+ 338us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.626622 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2730,"thread_start_us":79,"threads_started":1}
I20260621 05:19:08.628079 16505 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40959
I20260621 05:19:08.628192 16505 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:08.628378 16505 heartbeater.cc:511] Master 127.0.98.62:40959 requested a full tablet report, sending...
I20260621 05:19:08.629189 16317 ts_manager.cc:194] Registered new tserver with Master: 69587104581d4b80a1e970d6cd9ac179 (127.0.98.1:38499)
I20260621 05:19:08.630487 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57929
I20260621 05:19:08.631428 16505 heartbeater.cc:503] Master 127.0.98.62:40959 was elected leader, sending a full tablet report...
I20260621 05:19:08.635843 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.629837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38499 (local address 127.0.0.1:53498)
0621 05:19:08.631716 (+ 1879us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.631757 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:08.631824 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:08.632143 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:08.632145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:08.632147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:08.632206 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.632208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.633496 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.633497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.634618 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.634621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.635540 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.635541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.635646 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.635661 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 05:19:08.635731 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 05:19:08.635776 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1841,"thread_start_us":94,"threads_started":1}
I20260621 05:19:08.635861 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.630068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.0.98.1:38499)
0621 05:19:08.631959 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.631975 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.631985 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:19:08.632023 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.632025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.632042 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.632125 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:08.632288 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.633413 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.634705 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.635456 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.635529 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.635578 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.635756 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.635794 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1856,"thread_start_us":59,"threads_started":1}
I20260621 05:19:08.636574 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb552e446964177908457277d0895ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:18:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53498
I20260621 05:19:08.637136 16514 tablet_replica.cc:330] stopping tablet replica
I20260621 05:19:08.637255 16514 raft_consensus.cc:2229] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:08.637332 16514 raft_consensus.cc:2258] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:08.637661 16514 ts_tablet_manager.cc:1905] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:08.639724 16514 ts_tablet_manager.cc:1918] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:08.639814 16514 log.cc:1196] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal/wals/9cb552e446964177908457277d0895ce
I20260621 05:19:08.640043 16514 ts_tablet_manager.cc:1939] T 9cb552e446964177908457277d0895ce P 69587104581d4b80a1e970d6cd9ac179: Deleting consensus metadata
I20260621 05:19:08.640506 16302 catalog_manager.cc:4841] TS 69587104581d4b80a1e970d6cd9ac179 (127.0.98.1:38499): tablet 9cb552e446964177908457277d0895ce (table test-table [id=d54a74803def4bebae06e5d9dff32276]) successfully deleted
W20260621 05:19:08.705461 16509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:08.705667 16509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:08.705690 16509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:08.707024 16509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:08.707074 16509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:08.707101 16509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:19:08.708603 16509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35337
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:08.708796 16509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:08.708967 16509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:08.713660 16519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:08.713661 16522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:08.713699 16520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:08.713793 16509 server_base.cc:1034] running on GCE node
I20260621 05:19:08.714071 16509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:08.714258 16509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:08.715441 16509 hybrid_clock.cc:648] HybridClock initialized: now 1782019148715428 us; error 24 us; skew 500 ppm
Jun 21 05:19:08 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:08.718180 16509 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:19:08.718662 16509 webserver.cc:460] Webserver started at http://127.0.98.2:35337/ using document root <none> and password file <none>
I20260621 05:19:08.718883 16509 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:08.718976 16509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:08.720451 16509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:19:08.721117 16529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.721309 16509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:19:08.721411 16509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "dfbd3850162e4365a881511587fcf04d"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "c1d0687a0d6ca5a2cca5e6c4e717e2b6"
server_key_iv: "694f2cac3f9672b2d967549f01636794"
server_key_version: "encryptionkey@0"
I20260621 05:19:08.721776 16509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:08.742640 16509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:08.744906 16509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:08.745065 16509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:08.745306 16509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:08.745927 16509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:08.745997 16509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.746047 16509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:08.746070 16509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.751992 16509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36919
I20260621 05:19:08.752055 16642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36919 every 8 connection(s)
I20260621 05:19:08.752425 16509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:19:08.756867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 16509
I20260621 05:19:08.757143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39557
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40959
--webserver_port=41325
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:19:08 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:08.762835 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.753215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37503 (local address 127.0.98.62:40959)
0621 05:19:08.753293 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.753295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.753473 (+ 178us) server_negotiation.cc:407] Connection header received
0621 05:19:08.754162 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.754163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.754176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.754241 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:08.755408 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.756052 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.757216 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.757439 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.757525 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.759826 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:08.759828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:08.759828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:08.759835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:08.761264 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:08.761774 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:08.761776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:08.761776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:08.761838 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:08.762094 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:08.762094 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:08.762095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:08.762269 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:08.762335 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.762650 (+ 315us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.762737 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:19:08.763027 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.753194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:37503)
0621 05:19:08.753344 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.753385 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:08.753855 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:08.754318 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:08.754322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:08.755061 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:08.755255 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.755261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.756143 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.756144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.757118 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.757121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.757535 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.757536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.757667 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.758165 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:08.758180 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:08.759736 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:08.761374 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:08.761381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:08.761383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:08.761672 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:08.761904 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:08.761905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:08.761906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:08.762016 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:08.762335 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:08.762342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:08.762502 (+ 160us) client_negotiation.cc:769] Sending connection context
0621 05:19:08.762629 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 05:19:08.762779 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260621 05:19:08.763497 16643 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40959
I20260621 05:19:08.763595 16643 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:08.763797 16643 heartbeater.cc:511] Master 127.0.98.62:40959 requested a full tablet report, sending...
I20260621 05:19:08.764191 16317 ts_manager.cc:194] Registered new tserver with Master: dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
I20260621 05:19:08.764577 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37503
W20260621 05:19:08.837105 16646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:08.837328 16646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:08.837368 16646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:08.838482 16646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:08.838546 16646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:08.838591 16646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:19:08.840095 16646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41325
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:08.840354 16646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:08.840574 16646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:08.845489 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:08.845490 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:08.845515 16646 server_base.cc:1034] running on GCE node
W20260621 05:19:08.845535 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:08.845897 16646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:08.846133 16646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:08.847280 16646 hybrid_clock.cc:648] HybridClock initialized: now 1782019148847268 us; error 29 us; skew 500 ppm
Jun 21 05:19:08 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:08.849718 16646 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:19:08.850160 16646 webserver.cc:460] Webserver started at http://127.0.98.3:41325/ using document root <none> and password file <none>
I20260621 05:19:08.850358 16646 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:08.850407 16646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:08.851747 16646 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:19:08.852260 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.852418 16646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.852519 16646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "d3d3b9444b314c7cb8a00b43836a0d85"
format_stamp: "Formatted at 2026-06-21 05:18:48 on dist-test-slave-czm5"
server_key: "800856dee04937e1269c724f34187c96"
server_key_iv: "4e14e9dd25e12cd347f1b1ee8909cee5"
server_key_version: "encryptionkey@0"
I20260621 05:19:08.852838 16646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:08.860158 16646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:08.862102 16646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:08.862251 16646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:08.862481 16646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:08.862775 16646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:08.862831 16646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.862870 16646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:08.862897 16646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.868611 16646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39557
I20260621 05:19:08.868646 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39557 every 8 connection(s)
I20260621 05:19:08.868979 16646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:19:08.871996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 16646
Jun 21 05:19:08 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:08.879103 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.872390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.98.62:40959)
0621 05:19:08.872533 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.872535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.872630 (+ 95us) server_negotiation.cc:407] Connection header received
0621 05:19:08.872688 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.872689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.872698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.872749 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:08.873456 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.873901 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.874841 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.875039 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.875111 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.876363 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:08.876364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:08.876364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:08.876372 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:08.877885 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:08.878278 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:08.878279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:08.878279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:08.878345 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:08.878541 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:08.878542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:08.878543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:08.878653 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:08.878762 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.878853 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.878970 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260621 05:19:08.879110 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.869763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55381 (local address 127.0.98.62:40959)
0621 05:19:08.869844 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.869846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.870056 (+ 210us) server_negotiation.cc:407] Connection header received
0621 05:19:08.870551 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.870552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.870564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.870626 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:08.871518 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.872126 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.873658 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.873870 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.873955 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.875990 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:08.875991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:08.875991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:08.875999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:08.877634 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:08.878055 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:08.878056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:08.878056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:08.878102 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:08.878363 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:08.878363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:08.878364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:08.878490 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:08.878570 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.878853 (+ 283us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.879022 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":54}
I20260621 05:19:08.879220 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.869764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:55381)
0621 05:19:08.869940 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.869973 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:08.870446 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:08.870692 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:08.870696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:08.871225 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:08.871404 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.871408 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.872235 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.872237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.873567 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.873569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.873963 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.873964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.874096 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.874561 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:08.874569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:08.875896 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:08.877719 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:08.877724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:08.877726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:08.877978 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:08.878165 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:08.878166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:08.878166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:08.878301 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:08.878572 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:08.878577 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:08.878700 (+ 123us) client_negotiation.cc:769] Sending connection context
0621 05:19:08.878811 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 05:19:08.878970 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260621 05:19:08.879834 16775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40959
I20260621 05:19:08.879911 16775 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:08.880093 16775 heartbeater.cc:511] Master 127.0.98.62:40959 requested a full tablet report, sending...
I20260621 05:19:08.880458 16317 ts_manager.cc:194] Registered new tserver with Master: d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557)
I20260621 05:19:08.880826 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55381
I20260621 05:19:08.890439 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:19:08.892964 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.890792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.0.98.62:40959)
0621 05:19:08.890846 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.890847 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.890972 (+ 125us) server_negotiation.cc:407] Connection header received
0621 05:19:08.891017 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.891018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.891044 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.891087 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:19:08.891166 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.891600 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.892366 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.892543 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.892593 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.892695 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:19:08.892775 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:19:08.892828 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.892857 (+ 29us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.892873 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:19:08.893240 16317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:19:08.894774 16317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:19:08.896936 16439 tablet_service.cc:1467] Processing CreateTablet for tablet bf6f2930c542452dba700c6fd1caf279 (DEFAULT_TABLE table=test-table [id=8e74452a37b34283a58dfc91db1ddbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:08.897125 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf6f2930c542452dba700c6fd1caf279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:08.898554 16397 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Bootstrap starting.
I20260621 05:19:08.898847 16397 tablet_bootstrap.cc:654] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:08.899382 16397 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: No bootstrap required, opened a new log
I20260621 05:19:08.899468 16397 ts_tablet_manager.cc:1397] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:19:08.899574 16397 raft_consensus.cc:348] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.899629 16397 raft_consensus.cc:374] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:08.899689 16397 raft_consensus.cc:729] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69587104581d4b80a1e970d6cd9ac179, State: Initialized, Role: FOLLOWER
I20260621 05:19:08.899739 16397 consensus_queue.cc:260] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.899827 16397 ts_tablet_manager.cc:1428] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:08.900606 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.896537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36919 (local address 127.0.0.1:35294)
0621 05:19:08.897007 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.897017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:08.897060 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:08.897236 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:08.897237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:08.897238 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:08.897278 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.897279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.898310 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.898311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.899373 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.899375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.900368 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.900369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.900429 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.900433 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:08.900489 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:19:08.900526 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":88,"threads_started":1}
I20260621 05:19:08.900647 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.896799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.0.98.2:36919)
0621 05:19:08.897044 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.897048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.897059 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:19:08.897127 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.897128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.897143 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.897238 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:08.897354 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.898228 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.899469 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.900212 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.900282 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.900333 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.900506 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.900540 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":109,"threads_started":1}
I20260621 05:19:08.900854 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.896476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39557 (local address 127.0.0.1:33954)
0621 05:19:08.896534 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.896545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:08.896583 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:08.896867 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:08.896868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:08.896869 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:08.896908 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.896909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.898081 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.898082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.899130 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.899132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.900607 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.900608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.900669 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.900672 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:08.900730 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:19:08.900767 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 05:19:08.901065 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.896522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.0.98.3:39557)
0621 05:19:08.896688 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.896692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.896701 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:08.896738 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.896740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.896771 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.896853 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:08.896969 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.897996 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.899210 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.900521 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.900583 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.900634 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.900959 (+ 325us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.900996 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260621 05:19:08.901453 16577 tablet_service.cc:1467] Processing CreateTablet for tablet bf6f2930c542452dba700c6fd1caf279 (DEFAULT_TABLE table=test-table [id=8e74452a37b34283a58dfc91db1ddbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:08.901600 16709 tablet_service.cc:1467] Processing CreateTablet for tablet bf6f2930c542452dba700c6fd1caf279 (DEFAULT_TABLE table=test-table [id=8e74452a37b34283a58dfc91db1ddbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:08.901768 16577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf6f2930c542452dba700c6fd1caf279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:08.901839 16709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf6f2930c542452dba700c6fd1caf279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:08.904983 16785 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: Bootstrap starting.
I20260621 05:19:08.905592 16785 tablet_bootstrap.cc:654] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:08.906028 16785 log.cc:824] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:08.906271 16786 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: Bootstrap starting.
I20260621 05:19:08.906632 16785 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: No bootstrap required, opened a new log
I20260621 05:19:08.906684 16785 ts_tablet_manager.cc:1397] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:19:08.906849 16786 tablet_bootstrap.cc:654] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:08.907320 16786 log.cc:824] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:08.908120 16786 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: No bootstrap required, opened a new log
I20260621 05:19:08.908198 16786 ts_tablet_manager.cc:1397] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:19:08.908325 16785 raft_consensus.cc:348] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.908473 16785 raft_consensus.cc:374] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:08.908494 16785 raft_consensus.cc:729] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d3b9444b314c7cb8a00b43836a0d85, State: Initialized, Role: FOLLOWER
I20260621 05:19:08.908617 16785 consensus_queue.cc:260] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.908901 16785 ts_tablet_manager.cc:1428] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:19:08.908967 16775 heartbeater.cc:503] Master 127.0.98.62:40959 was elected leader, sending a full tablet report...
I20260621 05:19:08.910020 16786 raft_consensus.cc:348] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.910151 16786 raft_consensus.cc:374] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:08.910189 16786 raft_consensus.cc:729] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbd3850162e4365a881511587fcf04d, State: Initialized, Role: FOLLOWER
I20260621 05:19:08.910352 16786 consensus_queue.cc:260] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.910640 16786 ts_tablet_manager.cc:1428] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:19:08.910670 16643 heartbeater.cc:503] Master 127.0.98.62:40959 was elected leader, sending a full tablet report...
I20260621 05:19:08.922730 16511 raft_consensus.cc:482] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:08.922811 16511 raft_consensus.cc:504] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.923099 16511 leader_election.cc:288] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557), dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
I20260621 05:19:08.926627 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.923251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54951 (local address 127.0.98.3:39557)
0621 05:19:08.923292 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.923294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.923301 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:19:08.923375 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.923376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.923386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.923437 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:08.923639 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.924314 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.925720 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.926289 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.926360 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.926369 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.926533 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.926566 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:19:08.926627 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.923320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37537 (local address 127.0.98.2:36919)
0621 05:19:08.923361 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.923363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.923450 (+ 87us) server_negotiation.cc:407] Connection header received
0621 05:19:08.923547 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.923548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.923560 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.923625 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:08.923777 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.924366 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.925731 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.926284 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.926360 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.926369 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.926533 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.926566 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:19:08.926666 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.923209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39557 (local address 127.0.98.1:54951)
0621 05:19:08.923254 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.923264 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:08.923301 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:08.923456 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:08.923457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:08.923459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:08.923542 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.923545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.924412 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.924413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.925637 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.925639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.926370 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.926372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.926452 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.926458 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:19:08.926515 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:19:08.926562 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":47}
I20260621 05:19:08.926676 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.923253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36919 (local address 127.0.98.1:37537)
0621 05:19:08.923388 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.923397 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:08.923474 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:08.923652 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:08.923653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:08.923655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:08.923715 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.923716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.924434 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.924435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.925637 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:08.925639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.926372 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.926372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:08.926452 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.926458 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:19:08.926515 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:19:08.926562 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":31,"thread_start_us":82,"threads_started":1}
I20260621 05:19:08.926975 16729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf6f2930c542452dba700c6fd1caf279" candidate_uuid: "69587104581d4b80a1e970d6cd9ac179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" is_pre_election: true
I20260621 05:19:08.927042 16597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf6f2930c542452dba700c6fd1caf279" candidate_uuid: "69587104581d4b80a1e970d6cd9ac179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbd3850162e4365a881511587fcf04d" is_pre_election: true
I20260621 05:19:08.927134 16729 raft_consensus.cc:2454] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69587104581d4b80a1e970d6cd9ac179 in term 0.
I20260621 05:19:08.927181 16597 raft_consensus.cc:2454] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69587104581d4b80a1e970d6cd9ac179 in term 0.
I20260621 05:19:08.927309 16393 leader_election.cc:302] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179, d3d3b9444b314c7cb8a00b43836a0d85; no voters:
I20260621 05:19:08.927410 16511 raft_consensus.cc:2781] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:19:08.927469 16511 raft_consensus.cc:482] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:19:08.927502 16511 raft_consensus.cc:3037] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:08.928136 16511 raft_consensus.cc:504] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.928265 16511 leader_election.cc:288] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 1 election: Requested vote from peers d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557), dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
I20260621 05:19:08.928428 16597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf6f2930c542452dba700c6fd1caf279" candidate_uuid: "69587104581d4b80a1e970d6cd9ac179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbd3850162e4365a881511587fcf04d"
I20260621 05:19:08.928486 16597 raft_consensus.cc:3037] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:08.929069 16597 raft_consensus.cc:2454] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69587104581d4b80a1e970d6cd9ac179 in term 1.
I20260621 05:19:08.929185 16729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf6f2930c542452dba700c6fd1caf279" candidate_uuid: "69587104581d4b80a1e970d6cd9ac179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d3b9444b314c7cb8a00b43836a0d85"
I20260621 05:19:08.929247 16392 leader_election.cc:302] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179, dfbd3850162e4365a881511587fcf04d; no voters:
I20260621 05:19:08.929277 16729 raft_consensus.cc:3037] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:08.929322 16511 raft_consensus.cc:2781] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:08.929473 16511 raft_consensus.cc:686] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 LEADER]: Becoming Leader. State: Replica: 69587104581d4b80a1e970d6cd9ac179, State: Running, Role: LEADER
I20260621 05:19:08.929538 16511 consensus_queue.cc:237] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:08.930042 16729 raft_consensus.cc:2454] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69587104581d4b80a1e970d6cd9ac179 in term 1.
I20260621 05:19:08.930197 16317 catalog_manager.cc:5495] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69587104581d4b80a1e970d6cd9ac179 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "69587104581d4b80a1e970d6cd9ac179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } health_report { overall_health: HEALTHY } } }
I20260621 05:19:08.934989 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.932925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.0.98.1:38499)
0621 05:19:08.932976 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:08.932978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:08.932984 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:19:08.933011 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:08.933011 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:08.933027 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:08.933102 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:19:08.933190 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.933593 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.934256 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:08.934458 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:08.934534 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:08.934674 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:19:08.934759 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:19:08.934829 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:08.934899 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 05:19:08.934917 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:19:08.936297 16597 raft_consensus.cc:1261] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 1 FOLLOWER]: Refusing update from remote peer 69587104581d4b80a1e970d6cd9ac179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:08.936339 16729 raft_consensus.cc:1261] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 1 FOLLOWER]: Refusing update from remote peer 69587104581d4b80a1e970d6cd9ac179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:08.936578 16511 consensus_queue.cc:1035] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:08.936738 16792 consensus_queue.cc:1035] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:08.938694 16794 mvcc.cc:205] Tried to move back new op lower bound from 7299150434040860672 to 7299150434015825920. Current Snapshot: MvccSnapshot[applied={T|T < 7299150434040860672}]
I20260621 05:19:08.939677 16795 mvcc.cc:205] Tried to move back new op lower bound from 7299150434040860672 to 7299150434015825920. Current Snapshot: MvccSnapshot[applied={T|T < 7299150434040860672}]
I20260621 05:19:08.939903 16796 mvcc.cc:205] Tried to move back new op lower bound from 7299150434040860672 to 7299150434015825920. Current Snapshot: MvccSnapshot[applied={T|T < 7299150434040860672 or (T in {7299150434040860672})}]
I20260621 05:19:08.940840 16316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:08.940927 16316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:08.941561 16316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Sending DeleteTablet for 3 replicas of tablet bf6f2930c542452dba700c6fd1caf279
I20260621 05:19:08.941824 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet bf6f2930c542452dba700c6fd1caf279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53498
I20260621 05:19:08.941838 16577 tablet_service.cc:1514] Processing DeleteTablet for tablet bf6f2930c542452dba700c6fd1caf279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35294
I20260621 05:19:08.941915 16514 tablet_replica.cc:330] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: stopping tablet replica
I20260621 05:19:08.941964 16514 raft_consensus.cc:2229] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:19:08.942009 16709 tablet_service.cc:1514] Processing DeleteTablet for tablet bf6f2930c542452dba700c6fd1caf279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33954
I20260621 05:19:08.942040 16514 raft_consensus.cc:2258] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:08.942059 16804 tablet_replica.cc:330] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: stopping tablet replica
I20260621 05:19:08.942137 16804 raft_consensus.cc:2229] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:08.942214 16804 raft_consensus.cc:2258] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:08.942344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 16355
I20260621 05:19:08.942906 16805 tablet_replica.cc:330] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: stopping tablet replica
I20260621 05:19:08.943017 16805 raft_consensus.cc:2229] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:08.943106 16805 raft_consensus.cc:2258] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:08.943221 16804 ts_tablet_manager.cc:1905] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:08.943465 16805 ts_tablet_manager.cc:1905] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:08.944842 16805 ts_tablet_manager.cc:1918] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:08.944922 16805 log.cc:1196] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/bf6f2930c542452dba700c6fd1caf279
I20260621 05:19:08.944950 16804 ts_tablet_manager.cc:1918] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:08.945046 16804 log.cc:1196] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/bf6f2930c542452dba700c6fd1caf279
I20260621 05:19:08.945232 16805 ts_tablet_manager.cc:1939] T bf6f2930c542452dba700c6fd1caf279 P d3d3b9444b314c7cb8a00b43836a0d85: Deleting consensus metadata
I20260621 05:19:08.945393 16804 ts_tablet_manager.cc:1939] T bf6f2930c542452dba700c6fd1caf279 P dfbd3850162e4365a881511587fcf04d: Deleting consensus metadata
I20260621 05:19:08.945632 16303 catalog_manager.cc:4841] TS d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557): tablet bf6f2930c542452dba700c6fd1caf279 (table test-table [id=8e74452a37b34283a58dfc91db1ddbb3]) successfully deleted
I20260621 05:19:08.945812 16302 catalog_manager.cc:4841] TS dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919): tablet bf6f2930c542452dba700c6fd1caf279 (table test-table [id=8e74452a37b34283a58dfc91db1ddbb3]) successfully deleted
W20260621 05:19:08.948230 16302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38499 (error 108)
I20260621 05:19:08.948547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 16509
I20260621 05:19:08.948720 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:08.948561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38499 (local address 127.0.0.1:53516)
0621 05:19:08.948637 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:08.948681 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 05:19:08.949321 16302 catalog_manager.cc:4584] TS 69587104581d4b80a1e970d6cd9ac179 (127.0.98.1:38499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf6f2930c542452dba700c6fd1caf279: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38499: connect: Connection refused (error 111)
I20260621 05:19:08.953625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 16646
I20260621 05:19:08.958099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 16284
I20260621 05:19:28.963382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40959
--webserver_interface=127.0.98.62
--webserver_port=37043
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:29.039553 16807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:29.039784 16807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:29.039806 16807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:29.040947 16807 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:19:29.040987 16807 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:19:29.041000 16807 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:19:29.041013 16807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:29.041025 16807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:29.041080 16807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:29.041100 16807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:29.041122 16807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:29.042550 16807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:29.042796 16807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:29.043051 16807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:29.047971 16812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:29.048111 16813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:29.047986 16815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:29.048209 16807 server_base.cc:1034] running on GCE node
I20260621 05:19:29.048408 16807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:29.048646 16807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:29.049829 16807 hybrid_clock.cc:648] HybridClock initialized: now 1782019169049812 us; error 35 us; skew 500 ppm
Jun 21 05:19:29 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:29.052223 16807 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:19:29.052646 16807 webserver.cc:460] Webserver started at http://127.0.98.62:37043/ using document root <none> and password file <none>
I20260621 05:19:29.052830 16807 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:29.052870 16807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:29.054033 16807 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:29.054597 16822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.054764 16807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.054847 16807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "48f88a596aaf4183b48f1e0185cecf91"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "8d59efdfcc52e76dff0a62ebfff9808b"
server_key_iv: "15130cf68f9447c95d7289f78a663ea7"
server_key_version: "encryptionkey@0"
I20260621 05:19:29.055193 16807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:29.074249 16807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:29.075847 16807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:29.075986 16807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:29.079941 16807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40959
I20260621 05:19:29.080009 16874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40959 every 8 connection(s)
I20260621 05:19:29.080400 16807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:29.081785 16875 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:19:29.082429 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Bootstrap starting.
I20260621 05:19:29.088290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 16807
I20260621 05:19:29.088573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38499
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40959
--webserver_port=34757
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:19:29.090641 16875 log.cc:824] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:29.092713 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:19:29.092999 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Bootstrap complete.
I20260621 05:19:29.095146 16875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:19:29.095381 16875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f88a596aaf4183b48f1e0185cecf91, State: Initialized, Role: FOLLOWER
I20260621 05:19:29.095580 16875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:19:29.095696 16875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:29.095752 16875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:29.095803 16875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 2 FOLLOWER]: Advancing to term 3
I20260621 05:19:29.096783 16875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:19:29.096899 16875 leader_election.cc:302] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f88a596aaf4183b48f1e0185cecf91; no voters:
I20260621 05:19:29.097069 16875 leader_election.cc:288] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 05:19:29.097119 16879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 05:19:29.097276 16879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [term 3 LEADER]: Becoming Leader. State: Replica: 48f88a596aaf4183b48f1e0185cecf91, State: Running, Role: LEADER
I20260621 05:19:29.097317 16875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:29.097450 16879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } }
I20260621 05:19:29.097779 16881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f88a596aaf4183b48f1e0185cecf91. Latest consensus state: current_term: 3 leader_uuid: "48f88a596aaf4183b48f1e0185cecf91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } } }
I20260621 05:19:29.097792 16880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "48f88a596aaf4183b48f1e0185cecf91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f88a596aaf4183b48f1e0185cecf91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40959 } } }
I20260621 05:19:29.097898 16881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:29.097901 16880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:29.098096 16887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:29.098836 16887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:29.098929 16887 catalog_manager.cc:1259] Loaded cluster ID: bcadd3aff772440785db0d06127c9a0b
I20260621 05:19:29.099001 16887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:29.100284 16887 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:29.100455 16887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Loaded TSK: 0
I20260621 05:19:29.100498 16887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48f88a596aaf4183b48f1e0185cecf91: Loaded TSK: 1
F20260621 05:19:29.100718 16887 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7feb018349ad google::LogMessage::Fail() at ??:0
@ 0x7feb0183788a google::LogMessage::SendToLog() at ??:0
@ 0x7feb018344de google::LogMessage::Flush() at ??:0
@ 0x7feb01835a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7feb03fa1104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7feb03f9ed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7feb0978e1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7feb0978e48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7feb0978e594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7feb09754c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7feb0975ead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7feb037020d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7feb036f7954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7feb0455b6db start_thread at ??:0
@ 0x7feb0033261f clone at ??:0
W20260621 05:19:29.201064 16877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:29.201299 16877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:29.201328 16877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:29.202941 16877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:29.203003 16877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:29.203042 16877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:19:29.205051 16877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34757
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:29.205294 16877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:29.205562 16877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:29.211808 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:29.211766 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:29.211922 16877 server_base.cc:1034] running on GCE node
W20260621 05:19:29.211762 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:29.212288 16877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:29.212548 16877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:29.213717 16877 hybrid_clock.cc:648] HybridClock initialized: now 1782019169213705 us; error 30 us; skew 500 ppm
Jun 21 05:19:29 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:29.216195 16877 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:19:29.216666 16877 webserver.cc:460] Webserver started at http://127.0.98.1:34757/ using document root <none> and password file <none>
I20260621 05:19:29.216909 16877 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:29.217036 16877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:29.218423 16877 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:29.219204 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.219424 16877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:29.219532 16877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "69587104581d4b80a1e970d6cd9ac179"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "be877c3aa72f25f3e5e334adca602b40"
server_key_iv: "8ac836bd2b1b1a64c0bc2de0d1323813"
server_key_version: "encryptionkey@0"
I20260621 05:19:29.219972 16877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:29.229517 16877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:29.231695 16877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:29.231838 16877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:29.232057 16877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:29.232434 16933 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:19:29.234160 16877 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:19:29.234212 16877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:19:29.234265 16877 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:19:29.234838 16877 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:19:29.234886 16877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:19:29.234952 16933 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Bootstrap starting.
I20260621 05:19:29.241951 16877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38499
I20260621 05:19:29.241989 17040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38499 every 8 connection(s)
I20260621 05:19:29.242384 16877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:19:29.243495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 16877
I20260621 05:19:29.243836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36919
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40959
--webserver_port=35337
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:19:29.245919 16933 log.cc:824] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:29.248162 16933 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:19:29.248452 16933 tablet_bootstrap.cc:492] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Bootstrap complete.
I20260621 05:19:29.249368 16933 ts_tablet_manager.cc:1397] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260621 05:19:29.250383 16933 raft_consensus.cc:348] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:29.250636 16933 raft_consensus.cc:729] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69587104581d4b80a1e970d6cd9ac179, State: Initialized, Role: FOLLOWER
I20260621 05:19:29.250800 16933 consensus_queue.cc:260] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:29.251121 16933 ts_tablet_manager.cc:1428] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260621 05:19:29.326761 17043 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:19:29.243184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:51615)
0621 05:19:29.245127 (+ 1943us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.245179 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:29.245854 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:29.326556 (+ 80702us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
W20260621 05:19:29.326997 17041 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40959 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40959: Client connection negotiation failed: client connection to 127.0.98.62:40959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:19:29.327302 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:29.327187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:51491)
0621 05:19:29.327240 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.327275 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:19:29.327699 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:29.327571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:45207)
0621 05:19:29.327655 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.327681 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:19:29.327833 17041 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:19:29.339125 17044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:29.339334 17044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:29.339371 17044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:29.340518 17044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:29.340579 17044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:29.340621 17044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:19:29.342000 17044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35337
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:29.342240 17044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:29.342447 17044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:29.347244 17054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:29.347244 17052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:29.347401 17044 server_base.cc:1034] running on GCE node
W20260621 05:19:29.347249 17051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:29.347819 17044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:29.348100 17044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:29.349279 17044 hybrid_clock.cc:648] HybridClock initialized: now 1782019169349275 us; error 36 us; skew 500 ppm
Jun 21 05:19:29 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:29.351557 17044 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:19:29.352029 17044 webserver.cc:460] Webserver started at http://127.0.98.2:35337/ using document root <none> and password file <none>
I20260621 05:19:29.352262 17044 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:29.352330 17044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:29.353601 17044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:29.354164 17061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.354319 17044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.354413 17044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "dfbd3850162e4365a881511587fcf04d"
format_stamp: "Formatted at 2026-06-21 05:18:47 on dist-test-slave-czm5"
server_key: "c1d0687a0d6ca5a2cca5e6c4e717e2b6"
server_key_iv: "694f2cac3f9672b2d967549f01636794"
server_key_version: "encryptionkey@0"
I20260621 05:19:29.354753 17044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:29.372623 17044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:29.374357 17044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:29.374543 17044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:29.374776 17044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:29.375079 17044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:29.375154 17044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.375195 17044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:29.375223 17044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.381047 17044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36919
I20260621 05:19:29.381125 17174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36919 every 8 connection(s)
I20260621 05:19:29.381438 17044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:19:29.382467 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:29.382123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:45111)
0621 05:19:29.382267 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.382334 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260621 05:19:29.382889 17175 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40959 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40959: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
I20260621 05:19:29.383138 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:29.383058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:44875)
0621 05:19:29.383099 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.383122 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260621 05:19:29.383405 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:29.383336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:48481)
0621 05:19:29.383372 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.383393 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260621 05:19:29.383522 17175 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:19:29.389595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 17044
I20260621 05:19:29.389822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39557
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40959
--webserver_port=41325
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:29.466874 17178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:29.467052 17178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:29.467073 17178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:29.468248 17178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:29.468291 17178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:29.468317 17178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:19:29.469681 17178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41325
--tserver_master_addrs=127.0.98.62:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:29.469897 17178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:29.470098 17178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:19:29.474867 17178 server_base.cc:1034] running on GCE node
W20260621 05:19:29.474776 17183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:29.474776 17186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:29.474858 17184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:29.475246 17178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:29.475427 17178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:29.476560 17178 hybrid_clock.cc:648] HybridClock initialized: now 1782019169476511 us; error 63 us; skew 500 ppm
Jun 21 05:19:29 dist-test-slave-czm5 krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:29.478933 17178 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:19:29.479359 17178 webserver.cc:460] Webserver started at http://127.0.98.3:41325/ using document root <none> and password file <none>
I20260621 05:19:29.479604 17178 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:29.479736 17178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:29.480963 17178 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:29.481518 17193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.481698 17178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:29.481793 17178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "d3d3b9444b314c7cb8a00b43836a0d85"
format_stamp: "Formatted at 2026-06-21 05:18:48 on dist-test-slave-czm5"
server_key: "800856dee04937e1269c724f34187c96"
server_key_iv: "4e14e9dd25e12cd347f1b1ee8909cee5"
server_key_version: "encryptionkey@0"
I20260621 05:19:29.482131 17178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:29.500902 17178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:29.502691 17178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:29.502858 17178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:29.503073 17178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:29.503381 17178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:29.503435 17178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.503475 17178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:29.503505 17178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:29.509443 17178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39557
I20260621 05:19:29.509490 17306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39557 every 8 connection(s)
I20260621 05:19:29.509821 17178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:19:29.510799 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:29.510456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:35757)
0621 05:19:29.510600 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.510667 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
W20260621 05:19:29.511241 17307 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40959 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40959: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
I20260621 05:19:29.511539 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:29.511446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:42021)
0621 05:19:29.511496 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.511522 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:19:29.511857 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:29.511781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:39265)
0621 05:19:29.511821 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:29.511843 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 05:19:29.511984 17307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:19:29.514209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 17178
I20260621 05:19:30.328369 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:30.328139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:41083)
0621 05:19:30.328275 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:30.328311 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260621 05:19:30.384064 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:30.383879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:44911)
0621 05:19:30.383989 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:30.384023 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260621 05:19:30.512492 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:30.512299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:58829)
0621 05:19:30.512409 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:30.512445 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260621 05:19:30.608569 17314 raft_consensus.cc:482] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:30.608692 17314 raft_consensus.cc:504] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:30.609056 17314 leader_election.cc:288] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557), dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
Jun 21 05:19:30 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 21 05:19:30 dist-test-slave-czm5 krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260621 05:19:30.619179 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:30.609206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36919 (local address 127.0.98.1:37971)
0621 05:19:30.609325 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:30.609332 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:30.609367 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:30.610202 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:30.610205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:30.610553 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:30.610730 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:30.610736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:30.611876 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:30.611877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:30.612692 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:30.612694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:30.613228 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:30.613229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:30.613334 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:30.613875 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:30.613883 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:30.615798 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:30.617597 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:30.617598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:30.617599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:30.617872 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:30.618161 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:30.618162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:30.618162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:30.618281 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:30.618675 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:30.618681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:30.618820 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 05:19:30.618922 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 05:19:30.619092 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":98,"thread_start_us":49,"threads_started":1}
I20260621 05:19:30.619195 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:30.609331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37971 (local address 127.0.98.2:36919)
0621 05:19:30.609507 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:30.609515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:30.609543 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:19:30.609980 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:30.609982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:30.610021 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:30.610188 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:30.610883 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:30.611781 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:30.612774 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:30.613136 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:30.613241 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:30.615893 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:30.615897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:30.615897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:30.615908 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:30.617504 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:30.617957 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:30.617960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:30.617960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:30.618090 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:30.618358 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:30.618359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:30.618359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:30.618601 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:30.618694 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:30.618970 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 05:19:30.619079 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":55,"threads_started":1}
I20260621 05:19:30.619210 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:30.609331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60677 (local address 127.0.98.3:39557)
0621 05:19:30.609504 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:30.609514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:30.609543 (+ 29us) server_negotiation.cc:407] Connection header received
0621 05:19:30.609979 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:30.609982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:30.610021 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:30.610175 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:30.610865 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:30.611800 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:30.612785 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:30.613135 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:30.613255 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:30.615513 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:30.615517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:30.615517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:30.615527 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:30.617382 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:30.617977 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:30.617982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:30.617982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:30.618110 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:30.618352 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:30.618353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:30.618354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:30.618608 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:30.618697 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:30.618968 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 05:19:30.619088 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":57,"threads_started":1}
I20260621 05:19:30.619382 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:30.609196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39557 (local address 127.0.98.1:60677)
0621 05:19:30.609248 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:30.609259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:30.609326 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:30.610202 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:30.610205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:30.610552 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:30.610730 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:30.610736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:30.611895 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:30.611896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:30.612697 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:30.612699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:30.613235 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:30.613236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:30.613335 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:30.613928 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:30.613932 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:30.615402 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:30.617465 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:30.617471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:30.617474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:30.617883 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:30.618176 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:30.618177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:30.618177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:30.618282 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:30.618679 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:30.618685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:30.618819 (+ 134us) client_negotiation.cc:769] Sending connection context
0621 05:19:30.618921 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 05:19:30.619169 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":154}
W20260621 05:19:30.620584 16929 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
W20260621 05:19:30.620718 16928 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
I20260621 05:19:30.620802 16928 leader_election.cc:302] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179; no voters: d3d3b9444b314c7cb8a00b43836a0d85, dfbd3850162e4365a881511587fcf04d
I20260621 05:19:30.620926 17314 raft_consensus.cc:2726] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:19:31.329075 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:31.328867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:41509)
0621 05:19:31.328993 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:31.329031 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260621 05:19:31.384752 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:31.384557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:58123)
0621 05:19:31.384674 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:31.384717 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260621 05:19:31.513136 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:31.512928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:34389)
0621 05:19:31.513063 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:31.513097 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260621 05:19:32.329732 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:32.329511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:56139)
0621 05:19:32.329648 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:32.329685 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260621 05:19:32.385324 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:32.385156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:56545)
0621 05:19:32.385260 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:32.385290 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260621 05:19:32.513850 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:32.513565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:45839)
0621 05:19:32.513771 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:32.513809 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":61,"threads_started":1}
I20260621 05:19:32.797103 17324 raft_consensus.cc:482] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:32.797235 17324 raft_consensus.cc:504] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:32.797448 17324 leader_election.cc:288] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557), dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
W20260621 05:19:32.797906 16929 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
W20260621 05:19:32.797983 16928 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
I20260621 05:19:32.798079 16928 leader_election.cc:302] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179; no voters: d3d3b9444b314c7cb8a00b43836a0d85, dfbd3850162e4365a881511587fcf04d
I20260621 05:19:32.798157 17324 raft_consensus.cc:2726] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:19:33.330364 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:33.330152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:38207)
0621 05:19:33.330285 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:33.330320 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260621 05:19:33.386034 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:33.385774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:48521)
0621 05:19:33.385945 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:33.385996 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260621 05:19:33.514470 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:33.514295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:49801)
0621 05:19:33.514405 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:33.514435 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260621 05:19:34.331054 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:34.330857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:33233)
0621 05:19:34.330977 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:34.331010 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260621 05:19:34.386587 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:34.386419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:35195)
0621 05:19:34.386524 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:34.386556 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260621 05:19:34.515056 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:34.514876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:57863)
0621 05:19:34.514992 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:34.515021 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260621 05:19:35.296911 17331 raft_consensus.cc:482] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:35.297025 17331 raft_consensus.cc:504] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:35.297261 17331 leader_election.cc:288] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557), dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
W20260621 05:19:35.297577 16929 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
W20260621 05:19:35.297650 16928 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
I20260621 05:19:35.297706 16929 leader_election.cc:302] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179; no voters: d3d3b9444b314c7cb8a00b43836a0d85, dfbd3850162e4365a881511587fcf04d
I20260621 05:19:35.297819 17331 raft_consensus.cc:2726] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:19:35.331719 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:35.331432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:38485)
0621 05:19:35.331645 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:35.331678 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":52,"threads_started":1}
I20260621 05:19:35.387250 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:35.387084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:51637)
0621 05:19:35.387187 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:35.387218 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260621 05:19:35.515887 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:35.515602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:54563)
0621 05:19:35.515814 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:35.515847 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":158,"threads_started":1}
I20260621 05:19:36.332362 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:36.332164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:57993)
0621 05:19:36.332283 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:36.332316 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260621 05:19:36.387853 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:36.387678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:56229)
0621 05:19:36.387785 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:36.387817 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260621 05:19:36.516494 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:36.516315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:56277)
0621 05:19:36.516424 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:36.516453 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260621 05:19:36.802183 17338 raft_consensus.cc:482] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:36.802275 17338 raft_consensus.cc:504] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:36.802454 17338 leader_election.cc:288] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557), dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
W20260621 05:19:36.802848 16929 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
W20260621 05:19:36.802944 16928 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
I20260621 05:19:36.803015 16928 leader_election.cc:302] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179; no voters: d3d3b9444b314c7cb8a00b43836a0d85, dfbd3850162e4365a881511587fcf04d
I20260621 05:19:36.803107 17338 raft_consensus.cc:2726] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:19:37.332955 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:37.332761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:37197)
0621 05:19:37.332886 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:37.332912 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260621 05:19:37.388449 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:37.388245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:42279)
0621 05:19:37.388380 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:37.388413 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260621 05:19:37.517102 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:37.516922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:50685)
0621 05:19:37.517032 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:37.517059 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260621 05:19:38.333679 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:38.333421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:57601)
0621 05:19:38.333589 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:38.333632 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260621 05:19:38.389050 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:38.388841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:49723)
0621 05:19:38.388976 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:38.389007 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260621 05:19:38.517789 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:38.517556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:39243)
0621 05:19:38.517701 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:38.517741 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260621 05:19:39.334390 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:39.334136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:40201)
0621 05:19:39.334304 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:39.334344 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260621 05:19:39.389770 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:39.389516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:56299)
0621 05:19:39.389682 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:39.389724 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260621 05:19:39.518520 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:39.518258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:52073)
0621 05:19:39.518412 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:39.518458 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260621 05:19:40.335076 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:40.334851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:49501)
0621 05:19:40.335002 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:40.335031 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260621 05:19:40.390349 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:40.390169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:51535)
0621 05:19:40.390280 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:40.390314 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260621 05:19:40.519182 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:40.518986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:42093)
0621 05:19:40.519110 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:40.519142 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260621 05:19:41.336000 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:41.335789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:51435)
0621 05:19:41.335918 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:41.335956 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260621 05:19:41.390964 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:41.390764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:55919)
0621 05:19:41.390878 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:41.390913 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260621 05:19:41.519837 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:41.519589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:33705)
0621 05:19:41.519764 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:41.519798 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":117,"threads_started":1}
I20260621 05:19:42.336623 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:42.336413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:56429)
0621 05:19:42.336537 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:42.336570 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260621 05:19:42.391667 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:42.391421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:50037)
0621 05:19:42.391548 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:42.391584 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260621 05:19:42.520412 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:42.520199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:48411)
0621 05:19:42.520334 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:42.520368 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260621 05:19:43.240605 17357 raft_consensus.cc:482] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:43.240698 17357 raft_consensus.cc:504] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d3b9444b314c7cb8a00b43836a0d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "dfbd3850162e4365a881511587fcf04d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36919 } } peers { permanent_uuid: "69587104581d4b80a1e970d6cd9ac179" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } }
I20260621 05:19:43.240833 17357 leader_election.cc:288] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557), dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919)
W20260621 05:19:43.241252 16929 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d3b9444b314c7cb8a00b43836a0d85 (127.0.98.3:39557): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
W20260621 05:19:43.241344 16928 leader_election.cc:341] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbd3850162e4365a881511587fcf04d (127.0.98.2:36919): Not found: Tablet not found: bf6f2930c542452dba700c6fd1caf279
I20260621 05:19:43.241417 16928 leader_election.cc:302] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69587104581d4b80a1e970d6cd9ac179; no voters: d3d3b9444b314c7cb8a00b43836a0d85, dfbd3850162e4365a881511587fcf04d
I20260621 05:19:43.241498 17357 raft_consensus.cc:2726] T bf6f2930c542452dba700c6fd1caf279 P 69587104581d4b80a1e970d6cd9ac179 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:19:43.337281 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:43.337068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:37651)
0621 05:19:43.337186 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:43.337222 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260621 05:19:43.392290 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:43.392107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:41653)
0621 05:19:43.392218 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:43.392252 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260621 05:19:43.521112 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:43.520897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:40011)
0621 05:19:43.521016 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:43.521046 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260621 05:19:44.337976 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:44.337771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.1:42363)
0621 05:19:44.337898 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:44.337932 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260621 05:19:44.392956 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:44.392769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.2:57863)
0621 05:19:44.392881 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:44.392914 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260621 05:19:44.515539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 16877
I20260621 05:19:44.520339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 17044
I20260621 05:19:44.521960 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:44.521730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40959 (local address 127.0.98.3:43067)
0621 05:19:44.521865 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:44.521903 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260621 05:19:44.524955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 17178
2026-06-21T05:19:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58085 ms)
[----------] 30 tests from SecurityITest (251053 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:44 dist-test-slave-czm5 krb5kdc[17366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:44 dist-test-slave-czm5 krb5kdc[17366](info): set up 2 sockets
Jun 21 05:19:44 dist-test-slave-czm5 krb5kdc[17366](info): commencing operation
krb5kdc: starting...
W20260621 05:19:45.551282 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:19:45 dist-test-slave-czm5 krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:19:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:19:45.664417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35501
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:45.743870 17379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:45.744105 17379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:45.744143 17379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:45.745298 17379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:45.745380 17379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:45.745407 17379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:45.745438 17379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:45.745461 17379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:45.746906 17379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:45.747187 17379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:45.747426 17379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:45.752599 17384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:45.752615 17385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:45.752719 17387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:45.752740 17379 server_base.cc:1034] running on GCE node
I20260621 05:19:45.753158 17379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:45.753469 17379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:45.754639 17379 hybrid_clock.cc:648] HybridClock initialized: now 1782019185754627 us; error 46 us; skew 500 ppm
Jun 21 05:19:45 dist-test-slave-czm5 krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:45.757149 17379 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:19:45.757618 17379 webserver.cc:460] Webserver started at http://127.0.98.62:38073/ using document root <none> and password file <none>
I20260621 05:19:45.757858 17379 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:45.757927 17379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:45.758080 17379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:45.759106 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "4450b31542fd480da90ad6178147f63d"
format_stamp: "Formatted at 2026-06-21 05:19:45 on dist-test-slave-czm5"
server_key: "3181bb9b3d1ddf750e8085df7cbab617"
server_key_iv: "da5ae45c3a0b89cb6420ad1e1ca0b997"
server_key_version: "encryptionkey@0"
I20260621 05:19:45.759464 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "4450b31542fd480da90ad6178147f63d"
format_stamp: "Formatted at 2026-06-21 05:19:45 on dist-test-slave-czm5"
server_key: "3181bb9b3d1ddf750e8085df7cbab617"
server_key_iv: "da5ae45c3a0b89cb6420ad1e1ca0b997"
server_key_version: "encryptionkey@0"
I20260621 05:19:45.760794 17379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:19:45.761485 17394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:45.761659 17379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:45.761758 17379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "4450b31542fd480da90ad6178147f63d"
format_stamp: "Formatted at 2026-06-21 05:19:45 on dist-test-slave-czm5"
server_key: "3181bb9b3d1ddf750e8085df7cbab617"
server_key_iv: "da5ae45c3a0b89cb6420ad1e1ca0b997"
server_key_version: "encryptionkey@0"
I20260621 05:19:45.761845 17379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:45.770507 17379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:45.772814 17379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:45.772974 17379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:45.777081 17379 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35501
I20260621 05:19:45.777118 17446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35501 every 8 connection(s)
I20260621 05:19:45.777455 17379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:45.778249 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:45.778723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 17379
I20260621 05:19:45.778841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:45.779022 392 external_mini_cluster.cc:1428] Setting key 1bab91b11737f55f24aaaff556909c3d
I20260621 05:19:45.781330 17447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d: Bootstrap starting.
I20260621 05:19:45.781921 17447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:45.782241 17447 log.cc:824] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:45.782940 17447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d: No bootstrap required, opened a new log
I20260621 05:19:45.785343 17447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4450b31542fd480da90ad6178147f63d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35501 } }
I20260621 05:19:45.785470 17447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:45.785491 17447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4450b31542fd480da90ad6178147f63d, State: Initialized, Role: FOLLOWER
I20260621 05:19:45.785604 17447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4450b31542fd480da90ad6178147f63d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35501 } }
I20260621 05:19:45.785702 17447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:45.785734 17447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:45.785794 17447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 0 FOLLOWER]: Advancing to term 1
Jun 21 05:19:45 dist-test-slave-czm5 krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:45.786469 17447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4450b31542fd480da90ad6178147f63d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35501 } }
I20260621 05:19:45.786586 17447 leader_election.cc:302] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4450b31542fd480da90ad6178147f63d; no voters:
I20260621 05:19:45.786778 17447 leader_election.cc:288] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:45.786839 17452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:45.786988 17452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [term 1 LEADER]: Becoming Leader. State: Replica: 4450b31542fd480da90ad6178147f63d, State: Running, Role: LEADER
I20260621 05:19:45.787052 17447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:45.787072 17452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4450b31542fd480da90ad6178147f63d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35501 } }
I20260621 05:19:45.787382 17452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4450b31542fd480da90ad6178147f63d. Latest consensus state: current_term: 1 leader_uuid: "4450b31542fd480da90ad6178147f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4450b31542fd480da90ad6178147f63d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35501 } } }
I20260621 05:19:45.787395 17453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4450b31542fd480da90ad6178147f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4450b31542fd480da90ad6178147f63d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35501 } } }
I20260621 05:19:45.787470 17452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:45.787495 17453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:45.787734 17459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:45.788231 17459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:45.789964 17459 catalog_manager.cc:1347] Generated new cluster ID: ede0c585af2d49a68b5f82ae62d0deef
I20260621 05:19:45.790026 17459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:45.790923 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:45.779538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.0.98.62:35501)
0621 05:19:45.779752 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:45.779759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:45.779783 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:19:45.780166 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:45.780172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:45.780214 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:45.780328 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:45.781104 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:45.782075 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:45.783035 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:45.783502 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:45.783647 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:45.786891 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:45.786895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:45.786896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:45.786910 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:45.789208 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:45.789650 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:45.789653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:45.789654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:45.789746 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:45.789962 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:45.789963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:45.789963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:45.790292 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:45.790422 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:45.790606 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 05:19:45.790738 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
I20260621 05:19:45.802129 17459 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:45.802649 17459 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:45.805655 17459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d: Generated new TSK 0
I20260621 05:19:45.805827 17459 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:19:45.831616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35501
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:45.911563 17473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:45.911788 17473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:45.911813 17473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:45.912906 17473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:45.912937 17473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:45.912961 17473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:19:45.914304 17473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:45.914474 17473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:45.914633 17473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:19:45.919569 17473 server_base.cc:1034] running on GCE node
W20260621 05:19:45.919536 17481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:45.919540 17478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:45.919562 17479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:45.920049 17473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:45.920259 17473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:45.921412 17473 hybrid_clock.cc:648] HybridClock initialized: now 1782019185921403 us; error 26 us; skew 500 ppm
Jun 21 05:19:45 dist-test-slave-czm5 krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:45.923862 17473 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:19:45.924268 17473 webserver.cc:460] Webserver started at http://127.0.98.1:46173/ using document root <none> and password file <none>
I20260621 05:19:45.924489 17473 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:45.924532 17473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:45.924693 17473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:45.925717 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "ddd234655d914c14b627a399120a8378"
format_stamp: "Formatted at 2026-06-21 05:19:45 on dist-test-slave-czm5"
server_key: "16476b13d747d553373e9fa1853cb65e"
server_key_iv: "7e17d88754253e16489af09038136a78"
server_key_version: "encryptionkey@0"
I20260621 05:19:45.926074 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ddd234655d914c14b627a399120a8378"
format_stamp: "Formatted at 2026-06-21 05:19:45 on dist-test-slave-czm5"
server_key: "16476b13d747d553373e9fa1853cb65e"
server_key_iv: "7e17d88754253e16489af09038136a78"
server_key_version: "encryptionkey@0"
I20260621 05:19:45.927335 17473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:19:45.928081 17488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:45.928260 17473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:45.928346 17473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "ddd234655d914c14b627a399120a8378"
format_stamp: "Formatted at 2026-06-21 05:19:45 on dist-test-slave-czm5"
server_key: "16476b13d747d553373e9fa1853cb65e"
server_key_iv: "7e17d88754253e16489af09038136a78"
server_key_version: "encryptionkey@0"
I20260621 05:19:45.928412 17473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:45.947942 17473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:45.949744 17473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:45.949929 17473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:45.950155 17473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:45.950480 17473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:45.950531 17473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:45.950562 17473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:45.950582 17473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:45.956225 17473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36921
I20260621 05:19:45.956302 17601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36921 every 8 connection(s)
I20260621 05:19:45.956638 17473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:19:45 dist-test-slave-czm5 krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:45.966265 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:45.957393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59075 (local address 127.0.98.62:35501)
0621 05:19:45.957452 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:45.957454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:45.957675 (+ 221us) server_negotiation.cc:407] Connection header received
0621 05:19:45.958895 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:45.958897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:45.958908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:45.958967 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:45.959578 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:45.960138 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:45.961154 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:45.961399 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:45.961497 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:45.963612 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:45.963613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:45.963614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:45.963658 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:45.964886 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:45.965284 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:45.965285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:45.965286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:45.965361 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:45.965609 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:45.965610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:45.965610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:45.965726 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:45.965808 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:45.966074 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 05:19:45.966160 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:19:45.966432 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:45.957372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35501 (local address 127.0.98.1:59075)
0621 05:19:45.957545 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:45.957585 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:45.958792 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:45.959039 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:45.959042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:45.959303 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:45.959485 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:45.959489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:45.960235 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:45.960236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:45.961053 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:45.961055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:45.961509 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:45.961510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:45.961638 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:45.962158 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:45.962166 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:45.963519 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:45.964983 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:45.964991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:45.964993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:45.965207 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:45.965432 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:45.965433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:45.965434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:45.965542 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:45.965797 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:45.965803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:45.965943 (+ 140us) client_negotiation.cc:769] Sending connection context
0621 05:19:45.966054 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 05:19:45.966204 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260621 05:19:45.966692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 17473
I20260621 05:19:45.966846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:19:45.966959 17602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35501
I20260621 05:19:45.967048 392 external_mini_cluster.cc:1428] Setting key 3c6d4139fd6dff791d14b58baf169c74
I20260621 05:19:45.967053 17602 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:45.967232 17602 heartbeater.cc:511] Master 127.0.98.62:35501 requested a full tablet report, sending...
I20260621 05:19:45.967760 17410 ts_manager.cc:194] Registered new tserver with Master: ddd234655d914c14b627a399120a8378 (127.0.98.1:36921)
I20260621 05:19:45.968364 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59075
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:19:45.990309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35501
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:46.069893 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:46.070181 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:46.070225 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:46.071568 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:46.071611 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:46.071709 17607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:19:46.073100 17607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:46.073271 17607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:46.073444 17607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:46.078254 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.078284 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.078282 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.078497 17607 server_base.cc:1034] running on GCE node
I20260621 05:19:46.078696 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:46.078879 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:46.080046 17607 hybrid_clock.cc:648] HybridClock initialized: now 1782019186080032 us; error 28 us; skew 500 ppm
Jun 21 05:19:46 dist-test-slave-czm5 krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:46.082530 17607 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:19:46.082958 17607 webserver.cc:460] Webserver started at http://127.0.98.2:37771/ using document root <none> and password file <none>
I20260621 05:19:46.083151 17607 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:46.083192 17607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:46.083277 17607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:46.084303 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "c2d497db577e47c08b632d61bab3f87e"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "cfbd55a10f721d843a9f87b6377089b2"
server_key_iv: "0b92ace9fd398b560193f2bf04c3e5c9"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.084606 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c2d497db577e47c08b632d61bab3f87e"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "cfbd55a10f721d843a9f87b6377089b2"
server_key_iv: "0b92ace9fd398b560193f2bf04c3e5c9"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.085875 17607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:19:46.086489 17622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.086661 17607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:46.086760 17607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "c2d497db577e47c08b632d61bab3f87e"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "cfbd55a10f721d843a9f87b6377089b2"
server_key_iv: "0b92ace9fd398b560193f2bf04c3e5c9"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.086838 17607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:46.100805 17607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:46.103099 17607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:46.103262 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:46.103497 17607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:46.103869 17607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:46.103933 17607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.103968 17607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:46.103993 17607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.109743 17607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43103
I20260621 05:19:46.109808 17735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43103 every 8 connection(s)
I20260621 05:19:46.110146 17607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:19:46.115273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 17607
I20260621 05:19:46.115491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:19:46.115731 392 external_mini_cluster.cc:1428] Setting key e5977f8b255837ae10b5ad9c1d5aa398
Jun 21 05:19:46 dist-test-slave-czm5 krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:46.119541 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.110906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60193 (local address 127.0.98.62:35501)
0621 05:19:46.110968 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.110971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.111151 (+ 180us) server_negotiation.cc:407] Connection header received
0621 05:19:46.112099 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.112100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.112109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.112170 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:46.112748 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.113223 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.114164 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.114338 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.114383 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.116500 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:46.116501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:46.116502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:46.116508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:46.118189 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:46.118563 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:46.118565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:46.118565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:46.118635 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:46.118932 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:46.118933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:46.118933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:46.119038 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:46.119096 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.119415 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.119478 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:19:46.119845 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.110851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35501 (local address 127.0.98.2:60193)
0621 05:19:46.111014 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.111054 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.112011 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.112247 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.112251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.112490 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:46.112679 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.112683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.113305 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.113306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.114084 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.114086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.114401 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.114402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.114491 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.114927 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:46.114935 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:46.116422 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:46.118279 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:46.118285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:46.118286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:46.118493 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:46.118700 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:46.118701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:46.118701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:46.118847 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:46.119104 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:46.119109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:46.119277 (+ 168us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.119389 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.119556 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260621 05:19:46.120395 17736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35501
I20260621 05:19:46.120500 17736 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:46.120692 17736 heartbeater.cc:511] Master 127.0.98.62:35501 requested a full tablet report, sending...
I20260621 05:19:46.121026 17410 ts_manager.cc:194] Registered new tserver with Master: c2d497db577e47c08b632d61bab3f87e (127.0.98.2:43103)
I20260621 05:19:46.121392 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60193
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:19:46.140120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35501
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:46.219870 17741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:46.220069 17741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:46.220093 17741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:46.221184 17741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:46.221217 17741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:46.221243 17741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:19:46.222622 17741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:46.222793 17741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:46.222991 17741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:46.227937 17749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.227943 17746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.227977 17747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.228223 17741 server_base.cc:1034] running on GCE node
I20260621 05:19:46.228432 17741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:46.228696 17741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:46.229842 17741 hybrid_clock.cc:648] HybridClock initialized: now 1782019186229829 us; error 30 us; skew 500 ppm
Jun 21 05:19:46 dist-test-slave-czm5 krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:46.232270 17741 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:19:46.232745 17741 webserver.cc:460] Webserver started at http://127.0.98.3:37667/ using document root <none> and password file <none>
I20260621 05:19:46.232976 17741 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:46.233042 17741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:46.233187 17741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:46.234355 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "5adde76150d14d62a38804c33adb7a77"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "6d57bc5d6e43861db51c61d28b6f0fb8"
server_key_iv: "3008865e241e28a6adee0973468a31b4"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.234691 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5adde76150d14d62a38804c33adb7a77"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "6d57bc5d6e43861db51c61d28b6f0fb8"
server_key_iv: "3008865e241e28a6adee0973468a31b4"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.235984 17741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:19:46.236660 17756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.236855 17741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:46.236951 17741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "5adde76150d14d62a38804c33adb7a77"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "6d57bc5d6e43861db51c61d28b6f0fb8"
server_key_iv: "3008865e241e28a6adee0973468a31b4"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.237036 17741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:46.255263 17741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:46.257149 17741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:46.257321 17741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:46.257565 17741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:46.257892 17741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:46.257946 17741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.257984 17741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:46.258010 17741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.263978 17741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44137
I20260621 05:19:46.264032 17869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44137 every 8 connection(s)
I20260621 05:19:46.264372 17741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:19:46.264681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 17741
I20260621 05:19:46.264791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:19:46.264984 392 external_mini_cluster.cc:1428] Setting key 477d96774469ac379f364bf8a1452592
Jun 21 05:19:46 dist-test-slave-czm5 krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:46.274538 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.265167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52635 (local address 127.0.98.62:35501)
0621 05:19:46.265221 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.265224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.265551 (+ 327us) server_negotiation.cc:407] Connection header received
0621 05:19:46.266737 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.266738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.266750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.266826 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:46.267508 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.268323 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.269862 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.270072 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.270147 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.272216 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:46.272217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:46.272217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:46.272223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:46.273378 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:46.273736 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:46.273736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:46.273737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:46.273783 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:46.273992 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:46.273992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:46.273993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:46.274114 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:46.274166 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.274414 (+ 248us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.274473 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:19:46.274902 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.265170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35501 (local address 127.0.98.3:52635)
0621 05:19:46.265425 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.265465 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.266625 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.266920 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.266925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.267205 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:46.267408 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.267412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.268431 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.268433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.269762 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.269764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.270151 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.270152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.270287 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.270742 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:46.270750 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:46.272139 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:46.273444 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:46.273449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:46.273451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:46.273672 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:46.273838 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:46.273839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:46.273839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:46.273931 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:46.274170 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:46.274173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:46.274290 (+ 117us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.274386 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.274500 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260621 05:19:46.275403 17870 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35501
I20260621 05:19:46.275503 17870 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:46.275682 17870 heartbeater.cc:511] Master 127.0.98.62:35501 requested a full tablet report, sending...
I20260621 05:19:46.276062 17410 ts_manager.cc:194] Registered new tserver with Master: 5adde76150d14d62a38804c33adb7a77 (127.0.98.3:44137)
I20260621 05:19:46.276549 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:19:46.276561 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52635
I20260621 05:19:46.283272 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.277987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.0.98.62:35501)
0621 05:19:46.278049 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.278051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.278156 (+ 105us) server_negotiation.cc:407] Connection header received
0621 05:19:46.278196 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.278197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.278204 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.278250 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:46.278749 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.279218 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.279971 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.280142 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.280196 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.281212 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:46.281213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:46.281214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:46.281219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:46.282271 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:46.282555 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:46.282555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:46.282556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:46.282600 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:46.282838 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:46.282839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:46.282840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:46.283003 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:46.283079 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.283161 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.283218 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:19:46.284832 17410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:19:46.286240 17410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:19:46.292426 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.288291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.0.1:48890)
0621 05:19:46.288581 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.288624 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.288746 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.289062 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.289063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.289064 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.289111 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.289113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.290506 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.290507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.291547 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.291550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.292201 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.292202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.292251 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.292269 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.292340 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.292375 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":46,"threads_started":1}
I20260621 05:19:46.292501 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.288394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.0.98.1:36921)
0621 05:19:46.288567 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.288571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.288707 (+ 136us) server_negotiation.cc:407] Connection header received
0621 05:19:46.288875 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.288877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.288894 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.288944 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.289213 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.290398 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.291680 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.292136 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.292190 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.292227 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.292387 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.292425 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260621 05:19:46.293226 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.288308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44137 (local address 127.0.0.1:47264)
0621 05:19:46.288703 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.288713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.288773 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.289042 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.289045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.289048 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.289117 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.289119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.290203 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.290204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.291547 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.291550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.292987 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.292988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.293047 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.293051 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.293112 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.293151 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":6,"thread_start_us":52,"threads_started":1}
I20260621 05:19:46.293246 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.288395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.0.98.3:44137)
0621 05:19:46.288656 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.288661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.288880 (+ 219us) server_negotiation.cc:407] Connection header received
0621 05:19:46.288925 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.288927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.288943 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.289024 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.289223 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.290086 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.292256 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.292910 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.292965 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.293000 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.293140 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.293174 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":82,"threads_started":1}
I20260621 05:19:46.293171 17536 tablet_service.cc:1467] Processing CreateTablet for tablet 48bce26fa8104f47b27e85c58daf2d5a (DEFAULT_TABLE table=test-table [id=25ee47726c5546a89b2d679a88fd4bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:46.293557 17536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bce26fa8104f47b27e85c58daf2d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:46.294320 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.288488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43103 (local address 127.0.0.1:36656)
0621 05:19:46.288766 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.288776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.288815 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.289110 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.289111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.289113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.289252 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.289255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.290532 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.290533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.291554 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.291556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.294075 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.294076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.294149 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.294153 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.294216 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.294257 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":33,"threads_started":1}
I20260621 05:19:46.294239 17804 tablet_service.cc:1467] Processing CreateTablet for tablet 48bce26fa8104f47b27e85c58daf2d5a (DEFAULT_TABLE table=test-table [id=25ee47726c5546a89b2d679a88fd4bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:46.294576 17804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bce26fa8104f47b27e85c58daf2d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:46.294613 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.288673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.0.98.2:43103)
0621 05:19:46.288900 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.288903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.288912 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:46.288946 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.288948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.288963 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.289031 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.289338 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.290430 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.293028 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.293928 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.293993 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.294041 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.294494 (+ 453us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.294538 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":104,"threads_started":1}
I20260621 05:19:46.295137 17670 tablet_service.cc:1467] Processing CreateTablet for tablet 48bce26fa8104f47b27e85c58daf2d5a (DEFAULT_TABLE table=test-table [id=25ee47726c5546a89b2d679a88fd4bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:46.295420 17670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bce26fa8104f47b27e85c58daf2d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:46.296607 17884 tablet_bootstrap.cc:492] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378: Bootstrap starting.
I20260621 05:19:46.297207 17884 tablet_bootstrap.cc:654] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:46.297755 17885 tablet_bootstrap.cc:492] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: Bootstrap starting.
I20260621 05:19:46.297844 17884 log.cc:824] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:46.298388 17885 tablet_bootstrap.cc:654] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:46.298591 17887 tablet_bootstrap.cc:492] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: Bootstrap starting.
I20260621 05:19:46.298629 17884 tablet_bootstrap.cc:492] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378: No bootstrap required, opened a new log
I20260621 05:19:46.298692 17884 ts_tablet_manager.cc:1397] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:19:46.298918 17885 log.cc:824] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:46.299290 17887 tablet_bootstrap.cc:654] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:46.299705 17885 tablet_bootstrap.cc:492] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: No bootstrap required, opened a new log
I20260621 05:19:46.299779 17885 ts_tablet_manager.cc:1397] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:19:46.299917 17887 log.cc:824] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:46.300685 17887 tablet_bootstrap.cc:492] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: No bootstrap required, opened a new log
I20260621 05:19:46.300756 17887 ts_tablet_manager.cc:1397] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:19:46.300817 17884 raft_consensus.cc:348] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.300982 17884 raft_consensus.cc:374] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:46.301023 17884 raft_consensus.cc:729] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd234655d914c14b627a399120a8378, State: Initialized, Role: FOLLOWER
I20260621 05:19:46.301131 17884 consensus_queue.cc:260] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.301394 17884 ts_tablet_manager.cc:1428] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:19:46.301528 17602 heartbeater.cc:503] Master 127.0.98.62:35501 was elected leader, sending a full tablet report...
I20260621 05:19:46.301709 17885 raft_consensus.cc:348] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.301889 17885 raft_consensus.cc:374] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:46.301929 17885 raft_consensus.cc:729] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5adde76150d14d62a38804c33adb7a77, State: Initialized, Role: FOLLOWER
I20260621 05:19:46.302037 17885 consensus_queue.cc:260] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.302289 17885 ts_tablet_manager.cc:1428] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:19:46.302317 17870 heartbeater.cc:503] Master 127.0.98.62:35501 was elected leader, sending a full tablet report...
I20260621 05:19:46.302912 17887 raft_consensus.cc:348] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.303089 17887 raft_consensus.cc:374] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:46.303128 17887 raft_consensus.cc:729] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d497db577e47c08b632d61bab3f87e, State: Initialized, Role: FOLLOWER
I20260621 05:19:46.303234 17887 consensus_queue.cc:260] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.303479 17887 ts_tablet_manager.cc:1428] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:19:46.303511 17736 heartbeater.cc:503] Master 127.0.98.62:35501 was elected leader, sending a full tablet report...
I20260621 05:19:46.319693 17892 raft_consensus.cc:482] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:46.319849 17892 raft_consensus.cc:504] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.320214 17892 leader_election.cc:288] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddd234655d914c14b627a399120a8378 (127.0.98.1:36921), 5adde76150d14d62a38804c33adb7a77 (127.0.98.3:44137)
I20260621 05:19:46.322917 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.320374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44137 (local address 127.0.98.2:38443)
0621 05:19:46.320450 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.320461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.320502 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.320634 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.320634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.320635 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.320706 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.320707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.321321 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.321322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.322086 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.322088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.322645 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.322646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.322728 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.322731 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.322806 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.322846 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":4}
I20260621 05:19:46.322914 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.320443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38443 (local address 127.0.98.3:44137)
0621 05:19:46.320496 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.320497 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.320506 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:46.320567 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.320568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.320576 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.320630 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.320791 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.321241 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.322179 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.322557 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.322636 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.322646 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.322816 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.322854 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:19:46.323014 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.320423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32949 (local address 127.0.98.1:36921)
0621 05:19:46.320470 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.320472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.320625 (+ 153us) server_negotiation.cc:407] Connection header received
0621 05:19:46.320693 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.320695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.320705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.320758 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.320919 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.321360 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.322290 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.322652 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.322732 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.322741 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.322909 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.322944 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:19:46.323212 17824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bce26fa8104f47b27e85c58daf2d5a" candidate_uuid: "c2d497db577e47c08b632d61bab3f87e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5adde76150d14d62a38804c33adb7a77" is_pre_election: true
I20260621 05:19:46.323321 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.320402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.98.2:32949)
0621 05:19:46.320549 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.320558 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.320641 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.320763 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.320764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.320765 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.320832 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.320833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.321438 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.321439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.322198 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.322200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.322733 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.322734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.322808 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.322812 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.322901 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.322938 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260621 05:19:46.323367 17824 raft_consensus.cc:2454] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d497db577e47c08b632d61bab3f87e in term 0.
I20260621 05:19:46.323592 17624 leader_election.cc:302] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5adde76150d14d62a38804c33adb7a77, c2d497db577e47c08b632d61bab3f87e; no voters:
I20260621 05:19:46.323722 17892 raft_consensus.cc:2781] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:19:46.323772 17892 raft_consensus.cc:482] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:19:46.323799 17892 raft_consensus.cc:3037] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:46.323750 17556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bce26fa8104f47b27e85c58daf2d5a" candidate_uuid: "c2d497db577e47c08b632d61bab3f87e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd234655d914c14b627a399120a8378" is_pre_election: true
I20260621 05:19:46.323895 17556 raft_consensus.cc:2454] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d497db577e47c08b632d61bab3f87e in term 0.
I20260621 05:19:46.324522 17892 raft_consensus.cc:504] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.324656 17892 leader_election.cc:288] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [CANDIDATE]: Term 1 election: Requested vote from peers ddd234655d914c14b627a399120a8378 (127.0.98.1:36921), 5adde76150d14d62a38804c33adb7a77 (127.0.98.3:44137)
I20260621 05:19:46.324806 17556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bce26fa8104f47b27e85c58daf2d5a" candidate_uuid: "c2d497db577e47c08b632d61bab3f87e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd234655d914c14b627a399120a8378"
I20260621 05:19:46.324819 17824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bce26fa8104f47b27e85c58daf2d5a" candidate_uuid: "c2d497db577e47c08b632d61bab3f87e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5adde76150d14d62a38804c33adb7a77"
I20260621 05:19:46.324880 17556 raft_consensus.cc:3037] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:46.324918 17824 raft_consensus.cc:3037] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:46.325584 17824 raft_consensus.cc:2454] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d497db577e47c08b632d61bab3f87e in term 1.
I20260621 05:19:46.325589 17556 raft_consensus.cc:2454] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d497db577e47c08b632d61bab3f87e in term 1.
I20260621 05:19:46.325770 17623 leader_election.cc:302] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2d497db577e47c08b632d61bab3f87e, ddd234655d914c14b627a399120a8378; no voters:
I20260621 05:19:46.325865 17892 raft_consensus.cc:2781] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:46.326000 17892 raft_consensus.cc:686] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 1 LEADER]: Becoming Leader. State: Replica: c2d497db577e47c08b632d61bab3f87e, State: Running, Role: LEADER
I20260621 05:19:46.326089 17892 consensus_queue.cc:237] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } }
I20260621 05:19:46.326880 17410 catalog_manager.cc:5495] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e reported cstate change: term changed from 0 to 1, leader changed from <none> to c2d497db577e47c08b632d61bab3f87e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c2d497db577e47c08b632d61bab3f87e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2d497db577e47c08b632d61bab3f87e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } health_report { overall_health: HEALTHY } } }
I20260621 05:19:46.337568 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.335294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.0.98.2:43103)
0621 05:19:46.335345 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.335348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.335409 (+ 61us) server_negotiation.cc:407] Connection header received
0621 05:19:46.335456 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.335457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.335474 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.335533 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:19:46.335674 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.336135 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.336911 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.337081 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.337129 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.337253 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:19:46.337341 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:19:46.337400 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.337431 (+ 31us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.337447 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:19:46.339366 17556 raft_consensus.cc:1261] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 1 FOLLOWER]: Refusing update from remote peer c2d497db577e47c08b632d61bab3f87e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:46.339360 17824 raft_consensus.cc:1261] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 1 FOLLOWER]: Refusing update from remote peer c2d497db577e47c08b632d61bab3f87e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:46.339732 17894 consensus_queue.cc:1035] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5adde76150d14d62a38804c33adb7a77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:46.339861 17892 consensus_queue.cc:1035] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd234655d914c14b627a399120a8378" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:46.344796 17411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:46.344900 17411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:46.345685 17411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4450b31542fd480da90ad6178147f63d: Sending DeleteTablet for 3 replicas of tablet 48bce26fa8104f47b27e85c58daf2d5a
I20260621 05:19:46.346009 17804 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bce26fa8104f47b27e85c58daf2d5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47264
I20260621 05:19:46.346046 17536 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bce26fa8104f47b27e85c58daf2d5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48890
I20260621 05:19:46.346300 17906 tablet_replica.cc:330] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: stopping tablet replica
I20260621 05:19:46.346290 17670 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bce26fa8104f47b27e85c58daf2d5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36656
I20260621 05:19:46.346403 17906 raft_consensus.cc:2229] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:46.346477 17907 tablet_replica.cc:330] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378: stopping tablet replica
I20260621 05:19:46.346482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 17473
I20260621 05:19:46.346499 17906 raft_consensus.cc:2258] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:46.346563 17907 raft_consensus.cc:2229] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:46.346652 17907 raft_consensus.cc:2258] T 48bce26fa8104f47b27e85c58daf2d5a P ddd234655d914c14b627a399120a8378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:46.346933 17906 ts_tablet_manager.cc:1905] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:46.349063 17906 ts_tablet_manager.cc:1918] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:46.349128 17906 log.cc:1196] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/48bce26fa8104f47b27e85c58daf2d5a
I20260621 05:19:46.349433 17906 ts_tablet_manager.cc:1939] T 48bce26fa8104f47b27e85c58daf2d5a P 5adde76150d14d62a38804c33adb7a77: Deleting consensus metadata
I20260621 05:19:46.349804 17908 tablet_replica.cc:330] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: stopping tablet replica
I20260621 05:19:46.349870 17396 catalog_manager.cc:4841] TS 5adde76150d14d62a38804c33adb7a77 (127.0.98.3:44137): tablet 48bce26fa8104f47b27e85c58daf2d5a (table test-table [id=25ee47726c5546a89b2d679a88fd4bf7]) successfully deleted
I20260621 05:19:46.349900 17908 raft_consensus.cc:2229] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:19:46.350016 17908 raft_consensus.cc:2258] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:46.350462 17908 ts_tablet_manager.cc:1905] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:46.352372 17908 ts_tablet_manager.cc:1918] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:46.352445 17908 log.cc:1196] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/48bce26fa8104f47b27e85c58daf2d5a
I20260621 05:19:46.352691 17908 ts_tablet_manager.cc:1939] T 48bce26fa8104f47b27e85c58daf2d5a P c2d497db577e47c08b632d61bab3f87e: Deleting consensus metadata
I20260621 05:19:46.353056 17395 catalog_manager.cc:4841] TS c2d497db577e47c08b632d61bab3f87e (127.0.98.2:43103): tablet 48bce26fa8104f47b27e85c58daf2d5a (table test-table [id=25ee47726c5546a89b2d679a88fd4bf7]) successfully deleted
W20260621 05:19:46.353452 17395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36921 (error 108)
I20260621 05:19:46.353824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 17607
I20260621 05:19:46.354383 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.353767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.0.1:48900)
0621 05:19:46.354303 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.354344 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512}
W20260621 05:19:46.354483 17395 catalog_manager.cc:4584] TS ddd234655d914c14b627a399120a8378 (127.0.98.1:36921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48bce26fa8104f47b27e85c58daf2d5a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36921: connect: Connection refused (error 111)
I20260621 05:19:46.358889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 17741
I20260621 05:19:46.363160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 17379
2026-06-21T05:19:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1838 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T05:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:46Z Disabled control of system clock
I20260621 05:19:46.390048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36251
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:19:46.472210 17913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:46.472416 17913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:46.472440 17913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:46.473507 17913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:46.473543 17913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:46.473557 17913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:46.473569 17913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:46.473582 17913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:46.474831 17913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:46.475013 17913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:46.475232 17913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:46.479913 17918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.479921 17921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.480093 17913 server_base.cc:1034] running on GCE node
W20260621 05:19:46.479972 17919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.480510 17913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:46.480834 17913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:46.481994 17913 hybrid_clock.cc:648] HybridClock initialized: now 1782019186481991 us; error 37 us; skew 500 ppm
I20260621 05:19:46.483166 17913 webserver.cc:460] Webserver started at http://127.0.98.62:42983/ using document root <none> and password file <none>
I20260621 05:19:46.483424 17913 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:46.483493 17913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:46.483696 17913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:46.484725 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "3bef9098197f4ea3a6089b8306f3171c"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "776ef7894711c0c075fbe34684afb3b7"
server_key_iv: "454f023295e49369dcdeab2e1307e2b0"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.485085 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bef9098197f4ea3a6089b8306f3171c"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "776ef7894711c0c075fbe34684afb3b7"
server_key_iv: "454f023295e49369dcdeab2e1307e2b0"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.486414 17913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:19:46.487140 17927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.487313 17913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:46.487413 17913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "3bef9098197f4ea3a6089b8306f3171c"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "776ef7894711c0c075fbe34684afb3b7"
server_key_iv: "454f023295e49369dcdeab2e1307e2b0"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.487507 17913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:46.499722 17913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:46.500020 17913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:46.500174 17913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:46.504081 17913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36251
I20260621 05:19:46.504115 17979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36251 every 8 connection(s)
I20260621 05:19:46.504463 17913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:46.505399 17980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:46.507548 17980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c: Bootstrap starting.
I20260621 05:19:46.508167 17980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:46.508404 17980 log.cc:824] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:46.509008 17980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c: No bootstrap required, opened a new log
I20260621 05:19:46.510501 17980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bef9098197f4ea3a6089b8306f3171c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36251 } }
I20260621 05:19:46.510635 17980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:46.510658 17980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bef9098197f4ea3a6089b8306f3171c, State: Initialized, Role: FOLLOWER
I20260621 05:19:46.510758 17980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bef9098197f4ea3a6089b8306f3171c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36251 } }
I20260621 05:19:46.510810 17980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:46.510830 17980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:46.510864 17980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:46.511356 17980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bef9098197f4ea3a6089b8306f3171c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36251 } }
I20260621 05:19:46.511435 17980 leader_election.cc:302] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bef9098197f4ea3a6089b8306f3171c; no voters:
I20260621 05:19:46.511557 17980 leader_election.cc:288] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:46.511719 17983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:46.511855 17980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:46.511883 17983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [term 1 LEADER]: Becoming Leader. State: Replica: 3bef9098197f4ea3a6089b8306f3171c, State: Running, Role: LEADER
I20260621 05:19:46.511988 17983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bef9098197f4ea3a6089b8306f3171c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36251 } }
I20260621 05:19:46.512506 17984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bef9098197f4ea3a6089b8306f3171c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bef9098197f4ea3a6089b8306f3171c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36251 } } }
I20260621 05:19:46.512602 17984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:46.512929 17985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bef9098197f4ea3a6089b8306f3171c. Latest consensus state: current_term: 1 leader_uuid: "3bef9098197f4ea3a6089b8306f3171c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bef9098197f4ea3a6089b8306f3171c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36251 } } }
I20260621 05:19:46.512990 17992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:46.513016 17985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:46.513535 17992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:46.514220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 17913
I20260621 05:19:46.514320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:46.514518 392 external_mini_cluster.cc:1428] Setting key 5d44dda36d3beaea5fd1c96cae85999d
I20260621 05:19:46.515075 17992 catalog_manager.cc:1347] Generated new cluster ID: 262b77de8fbb4424b8556d37d141b6e6
I20260621 05:19:46.515127 17992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:46.520668 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.514981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.0.98.62:36251)
0621 05:19:46.515157 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.515165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.515354 (+ 189us) server_negotiation.cc:407] Connection header received
0621 05:19:46.515494 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.515501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.515567 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.515715 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:46.515948 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.517315 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.518428 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.519028 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.519138 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.519955 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:46.519960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:46.519960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:46.520160 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:46.520200 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:46.520213 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:46.520274 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.520397 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.520470 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260621 05:19:46.534096 17992 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:46.534853 17992 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:46.545563 17992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c: Generated new TSK 0
I20260621 05:19:46.545790 17992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:19:46.551122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36251
--builtin_ntp_servers=127.0.98.20:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:19:46.631929 18004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:46.632156 18004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:46.632193 18004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:46.633316 18004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:46.633381 18004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:46.633425 18004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:19:46.634758 18004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:46.635016 18004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:46.635303 18004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:46.640369 18009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.640385 18010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.640616 18004 server_base.cc:1034] running on GCE node
W20260621 05:19:46.640384 18012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.640895 18004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:46.641157 18004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:46.642328 18004 hybrid_clock.cc:648] HybridClock initialized: now 1782019186642315 us; error 27 us; skew 500 ppm
I20260621 05:19:46.643751 18004 webserver.cc:460] Webserver started at http://127.0.98.1:43885/ using document root <none> and password file <none>
I20260621 05:19:46.644003 18004 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:46.644062 18004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:46.644188 18004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:46.645548 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee757c39b3904c8083b2528750d78eec"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "2a7e7f294c47adecd56325aeadf64eaf"
server_key_iv: "afb05f09db29682afc81de601610d1de"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.645880 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee757c39b3904c8083b2528750d78eec"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "2a7e7f294c47adecd56325aeadf64eaf"
server_key_iv: "afb05f09db29682afc81de601610d1de"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.647087 18004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:19:46.647810 18018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.647959 18004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:46.648046 18004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "ee757c39b3904c8083b2528750d78eec"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "2a7e7f294c47adecd56325aeadf64eaf"
server_key_iv: "afb05f09db29682afc81de601610d1de"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.648099 18004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:46.657099 18004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:46.657346 18004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:46.657445 18004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:46.657614 18004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:46.657954 18004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:46.657989 18004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.658008 18004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:46.658049 18004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.663754 18004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40873
I20260621 05:19:46.663857 18131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40873 every 8 connection(s)
I20260621 05:19:46.664156 18004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:19:46.666092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 18004
I20260621 05:19:46.666213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:19:46.666467 392 external_mini_cluster.cc:1428] Setting key 00545503666d87c6ff490f8487dc6485
I20260621 05:19:46.667495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36251
--builtin_ntp_servers=127.0.98.20:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:19:46.670447 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.664891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41821 (local address 127.0.98.62:36251)
0621 05:19:46.664934 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.664936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.665198 (+ 262us) server_negotiation.cc:407] Connection header received
0621 05:19:46.665661 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.665662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.665673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.665728 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:46.666712 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.667305 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.668898 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.669221 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.669291 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.670127 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:46.670129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:46.670129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:46.670134 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:46.670152 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:46.670157 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:46.670215 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.670298 (+ 83us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.670327 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:19:46.670539 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.664854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36251 (local address 127.0.98.1:41821)
0621 05:19:46.665065 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.665106 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.665580 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.665803 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.665806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.665814 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:46.666598 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.666605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.667402 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.667403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.668785 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.668788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.669304 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.669305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.669416 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.669990 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:46.669990 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:46.670018 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:46.670040 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:46.670221 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:46.670224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:46.670227 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.670279 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.670358 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260621 05:19:46.671021 18132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36251
I20260621 05:19:46.671118 18132 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:46.671304 18132 heartbeater.cc:511] Master 127.0.98.62:36251 requested a full tablet report, sending...
I20260621 05:19:46.671965 17944 ts_manager.cc:194] Registered new tserver with Master: ee757c39b3904c8083b2528750d78eec (127.0.98.1:40873)
I20260621 05:19:46.672972 17944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41821
W20260621 05:19:46.748733 18135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:46.748924 18135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:46.748945 18135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:46.750020 18135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:46.750054 18135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:46.750078 18135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:19:46.751698 18135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:46.751873 18135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:46.752102 18135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:46.756960 18143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.756964 18140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.757046 18135 server_base.cc:1034] running on GCE node
W20260621 05:19:46.756989 18141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.757464 18135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:46.757717 18135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:46.758878 18135 hybrid_clock.cc:648] HybridClock initialized: now 1782019186758867 us; error 28 us; skew 500 ppm
I20260621 05:19:46.760133 18135 webserver.cc:460] Webserver started at http://127.0.98.2:36019/ using document root <none> and password file <none>
I20260621 05:19:46.760397 18135 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:46.760468 18135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:46.760618 18135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:46.761647 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd927c3860fd4b8d9d2f32a54158c676"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "323cd7115c903874a199b58ed28f8fc3"
server_key_iv: "2e56418095ecabddf4ad48149429e644"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.762001 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd927c3860fd4b8d9d2f32a54158c676"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "323cd7115c903874a199b58ed28f8fc3"
server_key_iv: "2e56418095ecabddf4ad48149429e644"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.763259 18135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:19:46.763996 18149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.764161 18135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.764258 18135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "fd927c3860fd4b8d9d2f32a54158c676"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "323cd7115c903874a199b58ed28f8fc3"
server_key_iv: "2e56418095ecabddf4ad48149429e644"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.764328 18135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:46.772488 18135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:46.772765 18135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:46.772923 18135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:46.773138 18135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:46.773445 18135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:46.773496 18135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.773535 18135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:46.773561 18135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.779309 18135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43117
I20260621 05:19:46.779346 18262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43117 every 8 connection(s)
I20260621 05:19:46.779724 18135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:19:46.783052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 18135
I20260621 05:19:46.783156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:19:46.783355 392 external_mini_cluster.cc:1428] Setting key 1816fd3b76ba125e8bb39fa4f8a5a5e9
I20260621 05:19:46.784360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36251
--builtin_ntp_servers=127.0.98.20:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:19:46.785543 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.780474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53575 (local address 127.0.98.62:36251)
0621 05:19:46.780533 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.780535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.780759 (+ 224us) server_negotiation.cc:407] Connection header received
0621 05:19:46.781295 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.781296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.781307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.781360 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:46.782056 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.782550 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.784111 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.784335 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.784416 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.785294 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:46.785295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:46.785296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:46.785300 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:46.785315 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:46.785319 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:46.785365 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.785438 (+ 73us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.785467 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:19:46.785756 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.780455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36251 (local address 127.0.98.2:53575)
0621 05:19:46.780649 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.780680 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.781182 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.781446 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.781450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.781459 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:46.781962 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.781966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.782675 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.782676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.784016 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.784019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.784426 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.784427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.784569 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.785136 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:46.785136 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:46.785163 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:46.785184 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:46.785364 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:46.785370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:46.785373 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.785441 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.785536 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260621 05:19:46.786173 18263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36251
I20260621 05:19:46.786271 18263 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:46.786453 18263 heartbeater.cc:511] Master 127.0.98.62:36251 requested a full tablet report, sending...
I20260621 05:19:46.786840 17944 ts_manager.cc:194] Registered new tserver with Master: fd927c3860fd4b8d9d2f32a54158c676 (127.0.98.2:43117)
I20260621 05:19:46.787357 17944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53575
W20260621 05:19:46.864755 18266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:46.864950 18266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:46.864971 18266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:46.866029 18266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:46.866061 18266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:46.866086 18266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:19:46.867514 18266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:46.867782 18266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:46.868085 18266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:46.872901 18271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.872941 18274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:46.872887 18272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:46.873057 18266 server_base.cc:1034] running on GCE node
I20260621 05:19:46.873271 18266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:46.873525 18266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:46.874683 18266 hybrid_clock.cc:648] HybridClock initialized: now 1782019186874679 us; error 36 us; skew 500 ppm
I20260621 05:19:46.875921 18266 webserver.cc:460] Webserver started at http://127.0.98.3:39759/ using document root <none> and password file <none>
I20260621 05:19:46.876160 18266 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:46.876228 18266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:46.876389 18266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:46.877480 18266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "5b21692b81b5452a87c75b2c6b3c3565"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "42e75da1939bf823f6b6a3a7bea5979f"
server_key_iv: "940894a90ebe047bdcb5d02c9a8697b5"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.877846 18266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5b21692b81b5452a87c75b2c6b3c3565"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "42e75da1939bf823f6b6a3a7bea5979f"
server_key_iv: "940894a90ebe047bdcb5d02c9a8697b5"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.879114 18266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:19:46.879904 18280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.880090 18266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:46.880188 18266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "5b21692b81b5452a87c75b2c6b3c3565"
format_stamp: "Formatted at 2026-06-21 05:19:46 on dist-test-slave-czm5"
server_key: "42e75da1939bf823f6b6a3a7bea5979f"
server_key_iv: "940894a90ebe047bdcb5d02c9a8697b5"
server_key_version: "encryptionkey@0"
I20260621 05:19:46.880270 18266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:46.905952 18266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:46.906280 18266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:46.906433 18266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:46.906662 18266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:46.907032 18266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:46.907095 18266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.907136 18266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:46.907164 18266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:46.913147 18266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37617
I20260621 05:19:46.913223 18393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37617 every 8 connection(s)
I20260621 05:19:46.913571 18266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:19:46.918874 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.914320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36811 (local address 127.0.98.62:36251)
0621 05:19:46.914370 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.914372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.914603 (+ 231us) server_negotiation.cc:407] Connection header received
0621 05:19:46.915170 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.915171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.915185 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.915236 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:46.916014 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.916513 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.917513 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.917713 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.917790 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.918573 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:46.918574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:46.918575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:46.918580 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:46.918595 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:46.918599 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:46.918669 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.918763 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.918793 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:19:46.919101 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.914290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36251 (local address 127.0.98.3:36811)
0621 05:19:46.914478 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.914512 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.915050 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.915305 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.915309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.915319 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:46.915910 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.915916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.916621 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.916622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.917412 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.917413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.917796 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.917797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.917934 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.918423 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:46.918423 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:46.918450 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:46.918471 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:46.918671 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:46.918676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:46.918680 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.918767 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.918874 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260621 05:19:46.919589 18394 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36251
I20260621 05:19:46.919714 18394 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:46.919909 18394 heartbeater.cc:511] Master 127.0.98.62:36251 requested a full tablet report, sending...
I20260621 05:19:46.920225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 18266
I20260621 05:19:46.920346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:19:46.920334 17944 ts_manager.cc:194] Registered new tserver with Master: 5b21692b81b5452a87c75b2c6b3c3565 (127.0.98.3:37617)
I20260621 05:19:46.920536 392 external_mini_cluster.cc:1428] Setting key 68cd778bb9b1d209dc9c898d948fbdb5
I20260621 05:19:46.920758 17944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36811
I20260621 05:19:46.921532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:19:46.925410 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.922835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.98.62:36251)
0621 05:19:46.922873 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.922875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.922987 (+ 112us) server_negotiation.cc:407] Connection header received
0621 05:19:46.923042 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.923043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.923054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.923091 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:46.923212 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.923715 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.924451 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.924621 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.924668 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.925224 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:46.925225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:46.925226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:46.925229 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:46.925241 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:46.925245 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:46.925290 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.925340 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.925358 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:19:46.927184 17943 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:19:46.928798 17943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:19:46.934796 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.930869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43117 (local address 127.0.0.1:55478)
0621 05:19:46.931044 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.931092 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.931158 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.931442 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.931445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.931448 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.931516 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.931518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.932894 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.932895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.933717 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.933720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.934510 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.934511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.934585 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.934597 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.934668 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.934714 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":33,"thread_start_us":81,"threads_started":1}
I20260621 05:19:46.934813 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.930984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.0.98.2:43117)
0621 05:19:46.931239 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.931245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.931257 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:19:46.931310 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.931313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.931332 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.931588 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.931667 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.932773 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.933809 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.934421 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.934493 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.934578 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.934694 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.934732 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
I20260621 05:19:46.935259 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.931083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40873 (local address 127.0.0.1:48006)
0621 05:19:46.931433 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.931444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.931495 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.931879 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.931885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.931887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.931921 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.931922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.932781 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.932782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.934040 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.934042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.935023 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.935024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.935087 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.935091 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.935141 (+ 50us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.935179 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":44,"thread_start_us":68,"threads_started":1}
I20260621 05:19:46.935359 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.930872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37617 (local address 127.0.0.1:35714)
0621 05:19:46.931306 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.931315 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.931367 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.931577 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.931577 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.931579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.931606 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.931607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.932862 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.932863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.934239 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.934241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.935117 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.935118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.935173 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.935177 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.935230 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.935269 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":73,"thread_start_us":74,"threads_started":1}
I20260621 05:19:46.935556 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.931197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.0.98.1:40873)
0621 05:19:46.931493 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.931497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.931508 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:19:46.931722 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.931724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.931741 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.931811 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.931996 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.932684 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.934121 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.934823 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.934908 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.934990 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.935403 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.935454 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":103,"threads_started":1}
I20260621 05:19:46.935842 18197 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6862853c4849309bdd3681870d31cf (DEFAULT_TABLE table=test-table [id=b8d15c0e7d584cf6abd2f934d0d50286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:46.936208 18197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6862853c4849309bdd3681870d31cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:46.936316 18066 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6862853c4849309bdd3681870d31cf (DEFAULT_TABLE table=test-table [id=b8d15c0e7d584cf6abd2f934d0d50286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:46.936655 18066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6862853c4849309bdd3681870d31cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:46.935385 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.931213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.0.98.3:37617)
0621 05:19:46.931389 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.931394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.931410 (+ 16us) server_negotiation.cc:407] Connection header received
0621 05:19:46.931455 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.931459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.931473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.931550 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.931716 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.932803 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.934322 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.935028 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.935116 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.935194 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.935270 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.935310 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260621 05:19:46.939036 18408 tablet_bootstrap.cc:492] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: Bootstrap starting.
I20260621 05:19:46.939481 18409 tablet_bootstrap.cc:492] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec: Bootstrap starting.
I20260621 05:19:46.939570 18328 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6862853c4849309bdd3681870d31cf (DEFAULT_TABLE table=test-table [id=b8d15c0e7d584cf6abd2f934d0d50286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:46.939750 18408 tablet_bootstrap.cc:654] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:46.939936 18328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6862853c4849309bdd3681870d31cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:46.940076 18408 log.cc:824] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:46.940135 18409 tablet_bootstrap.cc:654] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:46.940440 18409 log.cc:824] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:46.940883 18408 tablet_bootstrap.cc:492] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: No bootstrap required, opened a new log
I20260621 05:19:46.940979 18408 ts_tablet_manager.cc:1397] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:19:46.941244 18409 tablet_bootstrap.cc:492] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec: No bootstrap required, opened a new log
I20260621 05:19:46.941308 18409 ts_tablet_manager.cc:1397] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:19:46.942885 18412 tablet_bootstrap.cc:492] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: Bootstrap starting.
I20260621 05:19:46.943406 18412 tablet_bootstrap.cc:654] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:46.943753 18412 log.cc:824] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:46.943801 18408 raft_consensus.cc:348] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.943910 18409 raft_consensus.cc:348] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.944006 18408 raft_consensus.cc:374] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:46.944046 18409 raft_consensus.cc:374] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:46.944044 18408 raft_consensus.cc:729] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd927c3860fd4b8d9d2f32a54158c676, State: Initialized, Role: FOLLOWER
I20260621 05:19:46.944087 18409 raft_consensus.cc:729] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee757c39b3904c8083b2528750d78eec, State: Initialized, Role: FOLLOWER
I20260621 05:19:46.944186 18409 consensus_queue.cc:260] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.944195 18408 consensus_queue.cc:260] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.944453 18408 ts_tablet_manager.cc:1428] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260621 05:19:46.944453 18409 ts_tablet_manager.cc:1428] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260621 05:19:46.944569 18132 heartbeater.cc:503] Master 127.0.98.62:36251 was elected leader, sending a full tablet report...
I20260621 05:19:46.944593 18263 heartbeater.cc:503] Master 127.0.98.62:36251 was elected leader, sending a full tablet report...
I20260621 05:19:46.944751 18412 tablet_bootstrap.cc:492] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: No bootstrap required, opened a new log
I20260621 05:19:46.944851 18412 ts_tablet_manager.cc:1397] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:19:46.946935 18412 raft_consensus.cc:348] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.947077 18412 raft_consensus.cc:374] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:46.947120 18412 raft_consensus.cc:729] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b21692b81b5452a87c75b2c6b3c3565, State: Initialized, Role: FOLLOWER
I20260621 05:19:46.947232 18412 consensus_queue.cc:260] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.947460 18412 ts_tablet_manager.cc:1428] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:19:46.947544 18394 heartbeater.cc:503] Master 127.0.98.62:36251 was elected leader, sending a full tablet report...
I20260621 05:19:46.986295 18415 raft_consensus.cc:482] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:46.986436 18415 raft_consensus.cc:504] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.986826 18415 leader_election.cc:288] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b21692b81b5452a87c75b2c6b3c3565 (127.0.98.3:37617), ee757c39b3904c8083b2528750d78eec (127.0.98.1:40873)
I20260621 05:19:46.989910 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.987165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54623 (local address 127.0.98.1:40873)
0621 05:19:46.987208 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.987210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.987220 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:19:46.987273 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.987274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.987281 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.987321 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.987476 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.988173 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.989177 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.989527 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.989579 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.989613 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.989773 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.989826 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:19:46.989930 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.987024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58617 (local address 127.0.98.3:37617)
0621 05:19:46.987075 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:46.987077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:46.987086 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:46.987158 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:46.987159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:46.987171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:46.987219 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:46.987404 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.988179 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.989137 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.989528 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.989618 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.989626 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:46.989807 (+ 181us) server_negotiation.cc:299] Negotiation successful
0621 05:19:46.989846 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:19:46.989949 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.986956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37617 (local address 127.0.98.2:58617)
0621 05:19:46.987015 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.987026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.987070 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.987267 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.987268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.987269 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.987342 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.987344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.988252 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.988254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.989045 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.989047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.989638 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.989639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.989711 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.989714 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.989806 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.989846 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:19:46.989953 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:46.986991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40873 (local address 127.0.98.2:54623)
0621 05:19:46.987161 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:46.987171 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:46.987211 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:46.987343 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:46.987344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:46.987345 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:46.987404 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.987405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.988285 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.988287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.989072 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:46.989074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:46.989605 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:46.989606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:46.989680 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:46.989684 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:46.989735 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:19:46.989783 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
I20260621 05:19:46.990331 18348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6862853c4849309bdd3681870d31cf" candidate_uuid: "fd927c3860fd4b8d9d2f32a54158c676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b21692b81b5452a87c75b2c6b3c3565" is_pre_election: true
I20260621 05:19:46.990334 18086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6862853c4849309bdd3681870d31cf" candidate_uuid: "fd927c3860fd4b8d9d2f32a54158c676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee757c39b3904c8083b2528750d78eec" is_pre_election: true
I20260621 05:19:46.990501 18086 raft_consensus.cc:2454] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd927c3860fd4b8d9d2f32a54158c676 in term 0.
I20260621 05:19:46.990501 18348 raft_consensus.cc:2454] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd927c3860fd4b8d9d2f32a54158c676 in term 0.
I20260621 05:19:46.990731 18152 leader_election.cc:302] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee757c39b3904c8083b2528750d78eec, fd927c3860fd4b8d9d2f32a54158c676; no voters:
I20260621 05:19:46.990904 18415 raft_consensus.cc:2781] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:19:46.990994 18415 raft_consensus.cc:482] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:19:46.991029 18415 raft_consensus.cc:3037] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:46.991849 18415 raft_consensus.cc:504] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.991988 18415 leader_election.cc:288] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [CANDIDATE]: Term 1 election: Requested vote from peers 5b21692b81b5452a87c75b2c6b3c3565 (127.0.98.3:37617), ee757c39b3904c8083b2528750d78eec (127.0.98.1:40873)
I20260621 05:19:46.992125 18348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6862853c4849309bdd3681870d31cf" candidate_uuid: "fd927c3860fd4b8d9d2f32a54158c676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b21692b81b5452a87c75b2c6b3c3565"
I20260621 05:19:46.992146 18086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6862853c4849309bdd3681870d31cf" candidate_uuid: "fd927c3860fd4b8d9d2f32a54158c676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee757c39b3904c8083b2528750d78eec"
I20260621 05:19:46.992199 18348 raft_consensus.cc:3037] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:46.992234 18086 raft_consensus.cc:3037] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:46.992910 18348 raft_consensus.cc:2454] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd927c3860fd4b8d9d2f32a54158c676 in term 1.
I20260621 05:19:46.993041 18086 raft_consensus.cc:2454] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd927c3860fd4b8d9d2f32a54158c676 in term 1.
I20260621 05:19:46.993077 18150 leader_election.cc:302] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b21692b81b5452a87c75b2c6b3c3565, fd927c3860fd4b8d9d2f32a54158c676; no voters:
I20260621 05:19:46.993181 18415 raft_consensus.cc:2781] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:46.993331 18415 raft_consensus.cc:686] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 1 LEADER]: Becoming Leader. State: Replica: fd927c3860fd4b8d9d2f32a54158c676, State: Running, Role: LEADER
I20260621 05:19:46.993431 18415 consensus_queue.cc:237] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } }
I20260621 05:19:46.994215 17944 catalog_manager.cc:5495] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd927c3860fd4b8d9d2f32a54158c676 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fd927c3860fd4b8d9d2f32a54158c676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd927c3860fd4b8d9d2f32a54158c676" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260621 05:19:47.007357 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:47.005222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.0.98.2:43117)
0621 05:19:47.005282 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:47.005284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:47.005291 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:19:47.005325 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:47.005325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:47.005340 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:47.005399 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:19:47.005508 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:47.005974 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:47.006709 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:47.006890 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:47.006940 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:47.007066 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:19:47.007162 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:19:47.007221 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:47.007265 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 05:19:47.007284 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:19:47.009192 18086 raft_consensus.cc:1261] T 7e6862853c4849309bdd3681870d31cf P ee757c39b3904c8083b2528750d78eec [term 1 FOLLOWER]: Refusing update from remote peer fd927c3860fd4b8d9d2f32a54158c676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:47.009212 18348 raft_consensus.cc:1261] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 1 FOLLOWER]: Refusing update from remote peer fd927c3860fd4b8d9d2f32a54158c676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:47.009594 18418 consensus_queue.cc:1035] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee757c39b3904c8083b2528750d78eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:47.009727 18415 consensus_queue.cc:1035] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b21692b81b5452a87c75b2c6b3c3565" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:47.012097 18421 mvcc.cc:205] Tried to move back new op lower bound from 7299150589986988032 to 7299150589925691392. Current Snapshot: MvccSnapshot[applied={T|T < 7299150589986988032}]
I20260621 05:19:47.013260 18423 mvcc.cc:205] Tried to move back new op lower bound from 7299150589986988032 to 7299150589925691392. Current Snapshot: MvccSnapshot[applied={T|T < 7299150589986988032}]
I20260621 05:19:47.014717 17944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:47.014806 17944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:47.015503 17944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bef9098197f4ea3a6089b8306f3171c: Sending DeleteTablet for 3 replicas of tablet 7e6862853c4849309bdd3681870d31cf
I20260621 05:19:47.015847 18328 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e6862853c4849309bdd3681870d31cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:47 UTC) from {username='slave'} at 127.0.0.1:35714
I20260621 05:19:47.015899 18066 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e6862853c4849309bdd3681870d31cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:47 UTC) from {username='slave'} at 127.0.0.1:48006
I20260621 05:19:47.015920 18197 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e6862853c4849309bdd3681870d31cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:47 UTC) from {username='slave'} at 127.0.0.1:55478
I20260621 05:19:47.016135 18430 tablet_replica.cc:330] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: stopping tablet replica
I20260621 05:19:47.016234 18430 raft_consensus.cc:2229] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:47.016247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 18004
I20260621 05:19:47.016341 18430 raft_consensus.cc:2258] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:47.016841 18432 tablet_replica.cc:330] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: stopping tablet replica
I20260621 05:19:47.016937 18432 raft_consensus.cc:2229] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:19:47.017030 18432 raft_consensus.cc:2258] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:47.017516 18430 ts_tablet_manager.cc:1905] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:47.019541 18430 ts_tablet_manager.cc:1918] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:47.019611 18430 log.cc:1196] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/7e6862853c4849309bdd3681870d31cf
I20260621 05:19:47.019899 18430 ts_tablet_manager.cc:1939] T 7e6862853c4849309bdd3681870d31cf P 5b21692b81b5452a87c75b2c6b3c3565: Deleting consensus metadata
I20260621 05:19:47.020092 18432 ts_tablet_manager.cc:1905] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:47.020313 17928 catalog_manager.cc:4841] TS 5b21692b81b5452a87c75b2c6b3c3565 (127.0.98.3:37617): tablet 7e6862853c4849309bdd3681870d31cf (table test-table [id=b8d15c0e7d584cf6abd2f934d0d50286]) successfully deleted
I20260621 05:19:47.021530 18432 ts_tablet_manager.cc:1918] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:47.021605 18432 log.cc:1196] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/7e6862853c4849309bdd3681870d31cf
I20260621 05:19:47.021855 18432 ts_tablet_manager.cc:1939] T 7e6862853c4849309bdd3681870d31cf P fd927c3860fd4b8d9d2f32a54158c676: Deleting consensus metadata
I20260621 05:19:47.022253 17930 catalog_manager.cc:4841] TS fd927c3860fd4b8d9d2f32a54158c676 (127.0.98.2:43117): tablet 7e6862853c4849309bdd3681870d31cf (table test-table [id=b8d15c0e7d584cf6abd2f934d0d50286]) successfully deleted
W20260621 05:19:47.024650 17930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40873 (error 108)
I20260621 05:19:47.024907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 18135
I20260621 05:19:47.025269 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:47.025128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40873 (local address 127.0.0.1:48022)
0621 05:19:47.025198 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:47.025239 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 05:19:47.025604 17930 catalog_manager.cc:4584] TS ee757c39b3904c8083b2528750d78eec (127.0.98.1:40873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e6862853c4849309bdd3681870d31cf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40873: connect: Connection refused (error 111)
I20260621 05:19:47.029776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 18266
I20260621 05:19:47.034080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 17913
2026-06-21T05:19:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (670 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:47 dist-test-slave-czm5 krb5kdc[18436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:47 dist-test-slave-czm5 krb5kdc[18436](info): set up 2 sockets
Jun 21 05:19:47 dist-test-slave-czm5 krb5kdc[18436](info): commencing operation
krb5kdc: starting...
W20260621 05:19:48.055357 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:19:48.167874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37191
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:48.247635 18449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:48.247864 18449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:48.247900 18449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:48.249003 18449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:48.249068 18449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:48.249094 18449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:48.249125 18449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:48.249147 18449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:48.250597 18449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:48.250856 18449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:48.251065 18449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:48.255867 18455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:48.255857 18457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:48.255857 18454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:48.256083 18449 server_base.cc:1034] running on GCE node
I20260621 05:19:48.256289 18449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:48.256600 18449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:48.257771 18449 hybrid_clock.cc:648] HybridClock initialized: now 1782019188257758 us; error 27 us; skew 500 ppm
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:48.260195 18449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:19:48.260609 18449 webserver.cc:460] Webserver started at http://127.0.98.62:39337/ using document root <none> and password file <none>
I20260621 05:19:48.260840 18449 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:48.260914 18449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:48.261059 18449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:48.262043 18449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "94bfb0d0f4c14e5bb340920f4c1285cd"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "35c9e237a29a01ae7b6acc956b506f2a"
server_key_iv: "a7cd5e5c8c3afd6e0e0ceba5f17ab8d4"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.262378 18449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "94bfb0d0f4c14e5bb340920f4c1285cd"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "35c9e237a29a01ae7b6acc956b506f2a"
server_key_iv: "a7cd5e5c8c3afd6e0e0ceba5f17ab8d4"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.263739 18449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:19:48.264428 18464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.264601 18449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:48.264714 18449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "94bfb0d0f4c14e5bb340920f4c1285cd"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "35c9e237a29a01ae7b6acc956b506f2a"
server_key_iv: "a7cd5e5c8c3afd6e0e0ceba5f17ab8d4"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.264797 18449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:48.288996 18449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:48.290783 18449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:48.290951 18449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:48.294958 18449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37191
I20260621 05:19:48.295343 18449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:48.295358 18516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37191 every 8 connection(s)
I20260621 05:19:48.296136 18517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:48.298298 18517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd: Bootstrap starting.
I20260621 05:19:48.298835 18517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:48.299068 18517 log.cc:824] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:48.299751 18517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd: No bootstrap required, opened a new log
I20260621 05:19:48.301259 18517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94bfb0d0f4c14e5bb340920f4c1285cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } }
I20260621 05:19:48.301391 18517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:48.301414 18517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94bfb0d0f4c14e5bb340920f4c1285cd, State: Initialized, Role: FOLLOWER
I20260621 05:19:48.301542 18517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94bfb0d0f4c14e5bb340920f4c1285cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } }
I20260621 05:19:48.301616 18517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:48.301637 18517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:48.301666 18517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:48.302119 18517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94bfb0d0f4c14e5bb340920f4c1285cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } }
I20260621 05:19:48.302202 18517 leader_election.cc:302] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94bfb0d0f4c14e5bb340920f4c1285cd; no voters:
I20260621 05:19:48.302374 18517 leader_election.cc:288] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:48.302405 18520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:48.302659 18517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:48.302778 18520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [term 1 LEADER]: Becoming Leader. State: Replica: 94bfb0d0f4c14e5bb340920f4c1285cd, State: Running, Role: LEADER
I20260621 05:19:48.302871 18520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94bfb0d0f4c14e5bb340920f4c1285cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } }
I20260621 05:19:48.303251 18521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94bfb0d0f4c14e5bb340920f4c1285cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94bfb0d0f4c14e5bb340920f4c1285cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } }
I20260621 05:19:48.303344 18521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:48.303400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 18449
I20260621 05:19:48.303508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:48.303650 18522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94bfb0d0f4c14e5bb340920f4c1285cd. Latest consensus state: current_term: 1 leader_uuid: "94bfb0d0f4c14e5bb340920f4c1285cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94bfb0d0f4c14e5bb340920f4c1285cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } }
I20260621 05:19:48.303743 392 external_mini_cluster.cc:1428] Setting key 1fe3c81d88b02b845140e6bf417a4500
I20260621 05:19:48.303797 18522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:48.303962 18529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:48.304625 18529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:48.306038 18529 catalog_manager.cc:1347] Generated new cluster ID: 02c3973db365447eb03450c92168bad7
I20260621 05:19:48.306089 18529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:48.315047 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.304271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.0.98.62:37191)
0621 05:19:48.304451 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.304460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.304488 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:19:48.304599 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.304605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.304662 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.304797 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:48.305739 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.307067 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.308154 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.308716 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.308815 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.311530 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:48.311534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:48.311535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:48.311546 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:48.313468 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.313836 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.313839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.313839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.313940 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.314152 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.314153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.314153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.314464 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:48.314617 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.314712 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.314842 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260621 05:19:48.316195 18529 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:48.316676 18529 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:48.319013 18529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd: Generated new TSK 0
I20260621 05:19:48.319155 18529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:19:48.342041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37191
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:48.424073 18543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:48.424396 18543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:48.424450 18543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:48.425979 18543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:48.426036 18543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:48.426074 18543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:19:48.428216 18543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:48.428467 18543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:48.428723 18543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:48.433684 18551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:48.433650 18548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:48.433801 18543 server_base.cc:1034] running on GCE node
W20260621 05:19:48.433677 18549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:48.434103 18543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:48.434381 18543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:48.435525 18543 hybrid_clock.cc:648] HybridClock initialized: now 1782019188435501 us; error 40 us; skew 500 ppm
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:48.438174 18543 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:19:48.438651 18543 webserver.cc:460] Webserver started at http://127.0.98.1:41467/ using document root <none> and password file <none>
I20260621 05:19:48.438843 18543 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:48.438889 18543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:48.438977 18543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:48.440040 18543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "09b4c509fc744cea891e20751314e66b"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "820a6578c41e40ee1d756c801c5823dc"
server_key_iv: "2a34fbf1ed5354db919f4bc403472e48"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.440323 18543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "09b4c509fc744cea891e20751314e66b"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "820a6578c41e40ee1d756c801c5823dc"
server_key_iv: "2a34fbf1ed5354db919f4bc403472e48"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.441553 18543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:19:48.442255 18558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.442443 18543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:48.442541 18543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "09b4c509fc744cea891e20751314e66b"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "820a6578c41e40ee1d756c801c5823dc"
server_key_iv: "2a34fbf1ed5354db919f4bc403472e48"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.442636 18543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:48.455596 18543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:48.458024 18543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:48.458179 18543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:48.458390 18543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:48.458683 18543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:48.458734 18543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.458772 18543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:48.458798 18543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.464741 18543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39359
I20260621 05:19:48.464787 18671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39359 every 8 connection(s)
I20260621 05:19:48.465096 18543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:19:48.467918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 18543
I20260621 05:19:48.468024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:19:48.468231 392 external_mini_cluster.cc:1428] Setting key a8204f52ee346ac4375f46aa367209f6
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:48.474946 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.465710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37291 (local address 127.0.98.62:37191)
0621 05:19:48.465759 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.465761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.466018 (+ 257us) server_negotiation.cc:407] Connection header received
0621 05:19:48.466989 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.466990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.467003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.467055 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:48.467810 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.468551 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.469570 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.469782 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.469869 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.472019 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:48.472021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:48.472021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:48.472032 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:48.473655 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.474021 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.474022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.474023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.474084 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.474314 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.474315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.474316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.474450 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:48.474522 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.474747 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.474843 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:19:48.474964 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.465713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37191 (local address 127.0.98.1:37291)
0621 05:19:48.465905 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.465940 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:48.466862 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:48.467107 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:48.467110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:48.467468 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:48.467711 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.467717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.468650 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.468652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.469475 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.469477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.469879 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.469880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.470003 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.470504 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:48.470511 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:48.471926 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:48.473733 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:48.473739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:48.473741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:48.473947 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:48.474154 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:48.474154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:48.474155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:48.474244 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:48.474514 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:48.474518 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:48.474622 (+ 104us) client_negotiation.cc:769] Sending connection context
0621 05:19:48.474699 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 05:19:48.474808 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260621 05:19:48.475466 18672 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37191
I20260621 05:19:48.475565 18672 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:48.475771 18672 heartbeater.cc:511] Master 127.0.98.62:37191 requested a full tablet report, sending...
I20260621 05:19:48.476290 18481 ts_manager.cc:194] Registered new tserver with Master: 09b4c509fc744cea891e20751314e66b (127.0.98.1:39359)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260621 05:19:48.477011 18481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:19:48.492614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37191
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:48.573668 18677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:48.573858 18677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:48.573879 18677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:48.575054 18677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:48.575094 18677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:48.575120 18677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:19:48.576586 18677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:48.576905 18677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:48.577128 18677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:48.582094 18685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:48.582085 18683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:48.582070 18682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:48.582366 18677 server_base.cc:1034] running on GCE node
I20260621 05:19:48.582551 18677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:48.582742 18677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:48.583878 18677 hybrid_clock.cc:648] HybridClock initialized: now 1782019188583870 us; error 28 us; skew 500 ppm
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:48.586289 18677 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:19:48.586753 18677 webserver.cc:460] Webserver started at http://127.0.98.2:43715/ using document root <none> and password file <none>
I20260621 05:19:48.587026 18677 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:48.587082 18677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:48.587240 18677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:48.588304 18677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb039ae2c1704ceda59f678807ef22a9"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "0dbf00cd0295feb0bebb8a9639d991e3"
server_key_iv: "8115b8e7fdb2828026758f3d0ebe7c6c"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.588644 18677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb039ae2c1704ceda59f678807ef22a9"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "0dbf00cd0295feb0bebb8a9639d991e3"
server_key_iv: "8115b8e7fdb2828026758f3d0ebe7c6c"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.589927 18677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:19:48.590610 18692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.590785 18677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:48.590873 18677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "eb039ae2c1704ceda59f678807ef22a9"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "0dbf00cd0295feb0bebb8a9639d991e3"
server_key_iv: "8115b8e7fdb2828026758f3d0ebe7c6c"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.590943 18677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:48.609731 18677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:48.611490 18677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:48.611667 18677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:48.611897 18677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:48.612206 18677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:48.612237 18677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.612294 18677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:48.612321 18677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.618079 18677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43483
I20260621 05:19:48.618139 18805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43483 every 8 connection(s)
I20260621 05:19:48.618445 18677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:48.627012 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.619126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60907 (local address 127.0.98.62:37191)
0621 05:19:48.619197 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.619199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.619447 (+ 248us) server_negotiation.cc:407] Connection header received
0621 05:19:48.620417 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.620418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.620427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.620479 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:48.621041 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.621543 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.622458 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.622632 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.622680 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.624570 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:48.624571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:48.624571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:48.624578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:48.625932 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.626273 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.626273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.626274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.626323 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.626528 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.626529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.626529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.626625 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:48.626684 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.626894 (+ 210us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.626952 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:19:48.627197 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.619117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37191 (local address 127.0.98.2:60907)
0621 05:19:48.619288 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.619333 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:48.620326 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:48.620538 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:48.620541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:48.620787 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:48.620971 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.620974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.621616 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.621617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.622384 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.622386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.622700 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.622700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.622791 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.623214 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:48.623221 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:48.624490 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:48.625998 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:48.626003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:48.626005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:48.626211 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:48.626386 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:48.626387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:48.626387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:48.626476 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:48.626689 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:48.626693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:48.626795 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 05:19:48.626870 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:19:48.626974 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260621 05:19:48.627593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 18677
I20260621 05:19:48.627715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:19:48.627732 18806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37191
I20260621 05:19:48.627816 18806 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:48.627942 392 external_mini_cluster.cc:1428] Setting key 27952ae728bfd49a9491a0bc13f3bbc9
I20260621 05:19:48.628016 18806 heartbeater.cc:511] Master 127.0.98.62:37191 requested a full tablet report, sending...
I20260621 05:19:48.628384 18480 ts_manager.cc:194] Registered new tserver with Master: eb039ae2c1704ceda59f678807ef22a9 (127.0.98.2:43483)
I20260621 05:19:48.628791 18480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60907
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:19:48.650648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37191
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:48.728762 18811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:48.728967 18811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:48.728993 18811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:48.730103 18811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:48.730139 18811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:48.730165 18811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:19:48.731552 18811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:48.731739 18811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:48.731910 18811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:48.736724 18819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:48.736721 18816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:48.736703 18817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:48.737020 18811 server_base.cc:1034] running on GCE node
I20260621 05:19:48.737200 18811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:48.737439 18811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:48.738580 18811 hybrid_clock.cc:648] HybridClock initialized: now 1782019188738568 us; error 28 us; skew 500 ppm
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:48.740968 18811 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:19:48.741428 18811 webserver.cc:460] Webserver started at http://127.0.98.3:42727/ using document root <none> and password file <none>
I20260621 05:19:48.741650 18811 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:48.741717 18811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:48.741868 18811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:48.742882 18811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ce75bac4035421eb21c3d77cb264bc4"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "92dcb070d9da56be4abf3c6896c72980"
server_key_iv: "8ca1ec3cb25b1b8f96a01e971c7265f7"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.743284 18811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ce75bac4035421eb21c3d77cb264bc4"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "92dcb070d9da56be4abf3c6896c72980"
server_key_iv: "8ca1ec3cb25b1b8f96a01e971c7265f7"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.744665 18811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:19:48.745453 18826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.745617 18811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:48.745715 18811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "4ce75bac4035421eb21c3d77cb264bc4"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "92dcb070d9da56be4abf3c6896c72980"
server_key_iv: "8ca1ec3cb25b1b8f96a01e971c7265f7"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.745795 18811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:48.756340 18811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:48.758193 18811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:48.758350 18811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:48.758574 18811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:48.758868 18811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:48.758929 18811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.758970 18811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:48.758996 18811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.764653 18811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40929
I20260621 05:19:48.764707 18939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40929 every 8 connection(s)
I20260621 05:19:48.765049 18811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:19:48.765323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 18811
I20260621 05:19:48.765432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:19:48.765630 392 external_mini_cluster.cc:1428] Setting key b8f69a5af3f07c9460951642bced03aa
Jun 21 05:19:48 dist-test-slave-czm5 krb5kdc[18436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:48.773926 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.765803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49261 (local address 127.0.98.62:37191)
0621 05:19:48.765856 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.765858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.766087 (+ 229us) server_negotiation.cc:407] Connection header received
0621 05:19:48.767189 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.767190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.767202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.767250 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:48.767900 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.768434 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.769448 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.769653 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.769734 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.771738 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:48.771739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:48.771739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:48.771745 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:48.772893 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.773224 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.773224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.773224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.773269 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.773461 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.773462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.773462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.773553 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:48.773600 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.773811 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.773865 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:19:48.774080 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.765784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37191 (local address 127.0.98.3:49261)
0621 05:19:48.765966 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.766001 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:48.767078 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:48.767335 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:48.767342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:48.767574 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:48.767806 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.767811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.768521 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.768522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.769335 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.769337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.769743 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.769744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.769865 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.770257 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:48.770265 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:48.771567 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:48.772958 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:48.772964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:48.772965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:48.773163 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:48.773324 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:48.773324 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:48.773325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:48.773410 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:48.773603 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:48.773606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:48.773705 (+ 99us) client_negotiation.cc:769] Sending connection context
0621 05:19:48.773779 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:19:48.773893 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260621 05:19:48.774510 18940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37191
I20260621 05:19:48.774612 18940 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:48.774782 18940 heartbeater.cc:511] Master 127.0.98.62:37191 requested a full tablet report, sending...
I20260621 05:19:48.775154 18480 ts_manager.cc:194] Registered new tserver with Master: 4ce75bac4035421eb21c3d77cb264bc4 (127.0.98.3:40929)
I20260621 05:19:48.775511 18480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49261
I20260621 05:19:48.777350 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:19:48.784708 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.778606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.98.62:37191)
0621 05:19:48.778661 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.778663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.778810 (+ 147us) server_negotiation.cc:407] Connection header received
0621 05:19:48.778884 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.778885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.778894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.778968 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:48.779542 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.780146 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.780884 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.781091 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.781169 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.782381 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:48.782383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:48.782383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:48.782390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:48.783486 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.783882 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.783883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.783884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.783941 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:48.784160 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:48.784161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:48.784161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:48.784292 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:48.784402 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.784530 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.784589 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:19:48.786280 18480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:19:48.787828 18480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:19:48.794083 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.789813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39359 (local address 127.0.0.1:60060)
0621 05:19:48.790421 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.790429 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:48.790485 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:48.790748 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:48.790749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:48.790751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:48.790818 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.790820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.791798 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.791799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.792982 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.792985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.793805 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.793806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.793873 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.793884 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 05:19:48.793950 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:19:48.793995 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":86,"threads_started":1}
I20260621 05:19:48.794116 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.790018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.0.98.1:39359)
0621 05:19:48.790185 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.790190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.790490 (+ 300us) server_negotiation.cc:407] Connection header received
0621 05:19:48.790548 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.790550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.790568 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.790654 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:48.790907 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.791704 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.793065 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.793718 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.793785 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.793836 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.793976 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.794012 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260621 05:19:48.794337 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.789983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.0.98.2:43483)
0621 05:19:48.790156 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.790161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.790314 (+ 153us) server_negotiation.cc:407] Connection header received
0621 05:19:48.790406 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.790408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.790421 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.790516 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:48.790674 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.791915 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.793131 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.793785 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.793846 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.793898 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.794142 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.794190 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260621 05:19:48.794466 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.789816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40929 (local address 127.0.0.1:39942)
0621 05:19:48.790553 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.790562 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:48.790615 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:48.790998 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:48.790999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:48.791001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:48.791032 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.791034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.792266 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.792267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.793295 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.793297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.794221 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.794222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.794300 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.794304 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:48.794347 (+ 43us) client_negotiation.cc:240] Negotiation successful
0621 05:19:48.794392 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":88,"threads_started":1}
I20260621 05:19:48.794742 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.790053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.0.98.3:40929)
0621 05:19:48.790271 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.790275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.790715 (+ 440us) server_negotiation.cc:407] Connection header received
0621 05:19:48.790759 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.790761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.790774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.790861 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:48.791110 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.792161 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.793379 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.794051 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.794132 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.794184 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.794604 (+ 420us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.794645 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
I20260621 05:19:48.794880 18606 tablet_service.cc:1467] Processing CreateTablet for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26 (DEFAULT_TABLE table=test-table [id=ea0a909ca60f4ba6abe7c9d5078660aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:48.795241 18606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:48.795382 18874 tablet_service.cc:1467] Processing CreateTablet for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26 (DEFAULT_TABLE table=test-table [id=ea0a909ca60f4ba6abe7c9d5078660aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:48.795707 18874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:48.795719 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.789810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43483 (local address 127.0.0.1:40592)
0621 05:19:48.790195 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.790236 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:48.790317 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:48.790522 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:48.790525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:48.790528 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:48.790617 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.790618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.792024 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.792025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.793063 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.793066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.793944 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.793945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.793999 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.794001 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:19:48.794043 (+ 42us) client_negotiation.cc:240] Negotiation successful
0621 05:19:48.794070 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":25,"thread_start_us":95,"threads_started":1}
I20260621 05:19:48.796376 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26 (DEFAULT_TABLE table=test-table [id=ea0a909ca60f4ba6abe7c9d5078660aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:48.796674 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:48.798647 18954 tablet_bootstrap.cc:492] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b: Bootstrap starting.
I20260621 05:19:48.798897 18955 tablet_bootstrap.cc:492] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: Bootstrap starting.
I20260621 05:19:48.799295 18956 tablet_bootstrap.cc:492] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: Bootstrap starting.
I20260621 05:19:48.799413 18954 tablet_bootstrap.cc:654] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:48.799571 18955 tablet_bootstrap.cc:654] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:48.800028 18954 log.cc:824] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:48.800135 18956 tablet_bootstrap.cc:654] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:48.800185 18955 log.cc:824] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:48.800690 18956 log.cc:824] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:48.800925 18954 tablet_bootstrap.cc:492] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b: No bootstrap required, opened a new log
I20260621 05:19:48.800937 18955 tablet_bootstrap.cc:492] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: No bootstrap required, opened a new log
I20260621 05:19:48.800998 18955 ts_tablet_manager.cc:1397] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:19:48.800998 18954 ts_tablet_manager.cc:1397] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:19:48.801342 18956 tablet_bootstrap.cc:492] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: No bootstrap required, opened a new log
I20260621 05:19:48.801421 18956 ts_tablet_manager.cc:1397] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:19:48.802845 18955 raft_consensus.cc:348] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.802870 18954 raft_consensus.cc:348] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.802856 18956 raft_consensus.cc:348] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.803036 18955 raft_consensus.cc:374] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:48.803036 18954 raft_consensus.cc:374] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:48.803037 18956 raft_consensus.cc:374] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:48.803072 18955 raft_consensus.cc:729] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce75bac4035421eb21c3d77cb264bc4, State: Initialized, Role: FOLLOWER
I20260621 05:19:48.803072 18954 raft_consensus.cc:729] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b4c509fc744cea891e20751314e66b, State: Initialized, Role: FOLLOWER
I20260621 05:19:48.803081 18956 raft_consensus.cc:729] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb039ae2c1704ceda59f678807ef22a9, State: Initialized, Role: FOLLOWER
I20260621 05:19:48.803189 18955 consensus_queue.cc:260] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.803254 18954 consensus_queue.cc:260] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.803331 18956 consensus_queue.cc:260] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.803448 18955 ts_tablet_manager.cc:1428] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 05:19:48.803507 18954 ts_tablet_manager.cc:1428] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 05:19:48.803567 18940 heartbeater.cc:503] Master 127.0.98.62:37191 was elected leader, sending a full tablet report...
I20260621 05:19:48.803604 18672 heartbeater.cc:503] Master 127.0.98.62:37191 was elected leader, sending a full tablet report...
I20260621 05:19:48.804013 18956 ts_tablet_manager.cc:1428] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:19:48.804163 18806 heartbeater.cc:503] Master 127.0.98.62:37191 was elected leader, sending a full tablet report...
I20260621 05:19:48.816016 18961 raft_consensus.cc:482] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:48.816107 18961 raft_consensus.cc:504] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.816517 18961 leader_election.cc:288] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb039ae2c1704ceda59f678807ef22a9 (127.0.98.2:43483), 4ce75bac4035421eb21c3d77cb264bc4 (127.0.98.3:40929)
I20260621 05:19:48.819317 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.816737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41443 (local address 127.0.98.2:43483)
0621 05:19:48.816775 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.816777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.816786 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:48.816856 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.816857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.816870 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.816928 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:48.817101 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.817611 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.818586 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.818952 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.819006 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.819014 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.819210 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.819256 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:19:48.819344 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.816642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43483 (local address 127.0.98.1:41443)
0621 05:19:48.816711 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.816723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:48.816763 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:48.816953 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:48.816954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:48.816956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:48.817027 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.817029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.817686 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.817687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.818476 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.818479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.819031 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.819033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.819116 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.819119 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:48.819204 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 05:19:48.819248 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":21}
I20260621 05:19:48.819370 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.816752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43867 (local address 127.0.98.3:40929)
0621 05:19:48.816805 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.816808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.816933 (+ 125us) server_negotiation.cc:407] Connection header received
0621 05:19:48.816964 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.816964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.816974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.817039 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:48.817206 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.817659 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.818586 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.818962 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.819049 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.819059 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.819248 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.819295 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 05:19:48.819579 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.816693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40929 (local address 127.0.98.1:43867)
0621 05:19:48.816858 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.816868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:48.816921 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:48.817050 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:48.817050 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:48.817051 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:48.817141 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.817143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.817748 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.817749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.818483 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:48.818484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.819065 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.819066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:48.819139 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.819142 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:48.819445 (+ 303us) client_negotiation.cc:240] Negotiation successful
0621 05:19:48.819487 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260621 05:19:48.819834 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85db7dca2b9746f5a0aaf6f6f3b8ef26" candidate_uuid: "09b4c509fc744cea891e20751314e66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb039ae2c1704ceda59f678807ef22a9" is_pre_election: true
I20260621 05:19:48.819927 18894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85db7dca2b9746f5a0aaf6f6f3b8ef26" candidate_uuid: "09b4c509fc744cea891e20751314e66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce75bac4035421eb21c3d77cb264bc4" is_pre_election: true
I20260621 05:19:48.819964 18760 raft_consensus.cc:2454] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09b4c509fc744cea891e20751314e66b in term 0.
I20260621 05:19:48.820045 18894 raft_consensus.cc:2454] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09b4c509fc744cea891e20751314e66b in term 0.
I20260621 05:19:48.820204 18561 leader_election.cc:302] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b4c509fc744cea891e20751314e66b, eb039ae2c1704ceda59f678807ef22a9; no voters:
I20260621 05:19:48.820333 18961 raft_consensus.cc:2781] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:19:48.820402 18961 raft_consensus.cc:482] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:19:48.820420 18961 raft_consensus.cc:3037] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:48.821103 18961 raft_consensus.cc:504] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.821237 18961 leader_election.cc:288] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [CANDIDATE]: Term 1 election: Requested vote from peers eb039ae2c1704ceda59f678807ef22a9 (127.0.98.2:43483), 4ce75bac4035421eb21c3d77cb264bc4 (127.0.98.3:40929)
I20260621 05:19:48.821422 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85db7dca2b9746f5a0aaf6f6f3b8ef26" candidate_uuid: "09b4c509fc744cea891e20751314e66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb039ae2c1704ceda59f678807ef22a9"
I20260621 05:19:48.821492 18894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85db7dca2b9746f5a0aaf6f6f3b8ef26" candidate_uuid: "09b4c509fc744cea891e20751314e66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce75bac4035421eb21c3d77cb264bc4"
I20260621 05:19:48.821520 18760 raft_consensus.cc:3037] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:48.821552 18894 raft_consensus.cc:3037] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:48.822382 18760 raft_consensus.cc:2454] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09b4c509fc744cea891e20751314e66b in term 1.
I20260621 05:19:48.822382 18894 raft_consensus.cc:2454] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09b4c509fc744cea891e20751314e66b in term 1.
I20260621 05:19:48.822544 18561 leader_election.cc:302] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b4c509fc744cea891e20751314e66b, eb039ae2c1704ceda59f678807ef22a9; no voters:
I20260621 05:19:48.822681 18961 raft_consensus.cc:2781] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:48.822855 18961 raft_consensus.cc:686] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [term 1 LEADER]: Becoming Leader. State: Replica: 09b4c509fc744cea891e20751314e66b, State: Running, Role: LEADER
I20260621 05:19:48.822960 18961 consensus_queue.cc:237] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260621 05:19:48.823769 18481 catalog_manager.cc:5495] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b reported cstate change: term changed from 0 to 1, leader changed from <none> to 09b4c509fc744cea891e20751314e66b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "09b4c509fc744cea891e20751314e66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09b4c509fc744cea891e20751314e66b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } health_report { overall_health: UNKNOWN } } }
I20260621 05:19:48.831336 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.828949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.0.98.1:39359)
0621 05:19:48.828993 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.828995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.829004 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:48.829083 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.829084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.829102 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.829168 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:19:48.829313 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.829773 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.830533 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:48.830739 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:48.830823 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:48.830986 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:19:48.831081 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:19:48.831198 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:48.831229 (+ 31us) server_negotiation.cc:299] Negotiation successful
0621 05:19:48.831240 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:19:48.832966 18894 raft_consensus.cc:1261] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 1 FOLLOWER]: Refusing update from remote peer 09b4c509fc744cea891e20751314e66b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:48.832993 18760 raft_consensus.cc:1261] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 1 FOLLOWER]: Refusing update from remote peer 09b4c509fc744cea891e20751314e66b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:48.833328 18964 consensus_queue.cc:1035] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce75bac4035421eb21c3d77cb264bc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:48.833453 18961 consensus_queue.cc:1035] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 09b4c509fc744cea891e20751314e66b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb039ae2c1704ceda59f678807ef22a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:48.835595 18968 mvcc.cc:205] Tried to move back new op lower bound from 7299150597457457152 to 7299150597419417600. Current Snapshot: MvccSnapshot[applied={T|T < 7299150597457457152}]
I20260621 05:19:48.836084 18969 mvcc.cc:205] Tried to move back new op lower bound from 7299150597457457152 to 7299150597419417600. Current Snapshot: MvccSnapshot[applied={T|T < 7299150597457457152}]
I20260621 05:19:48.838428 18481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:48.838542 18481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:48.839293 18481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94bfb0d0f4c14e5bb340920f4c1285cd: Sending DeleteTablet for 3 replicas of tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26
I20260621 05:19:48.839656 18606 tablet_service.cc:1514] Processing DeleteTablet for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60060
I20260621 05:19:48.839655 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40592
I20260621 05:19:48.839743 18874 tablet_service.cc:1514] Processing DeleteTablet for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39942
I20260621 05:19:48.839882 18978 tablet_replica.cc:330] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: stopping tablet replica
I20260621 05:19:48.839972 18978 raft_consensus.cc:2229] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:48.840071 18978 raft_consensus.cc:2258] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:48.840262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 18543
I20260621 05:19:48.841095 18978 ts_tablet_manager.cc:1905] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:48.840209 18979 tablet_replica.cc:330] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: stopping tablet replica
I20260621 05:19:48.841457 18979 raft_consensus.cc:2229] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:48.841606 18979 raft_consensus.cc:2258] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:48.842043 18979 ts_tablet_manager.cc:1905] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:48.842561 18978 ts_tablet_manager.cc:1918] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:48.842643 18978 log.cc:1196] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/85db7dca2b9746f5a0aaf6f6f3b8ef26
I20260621 05:19:48.842895 18978 ts_tablet_manager.cc:1939] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P eb039ae2c1704ceda59f678807ef22a9: Deleting consensus metadata
I20260621 05:19:48.843272 18467 catalog_manager.cc:4841] TS eb039ae2c1704ceda59f678807ef22a9 (127.0.98.2:43483): tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26 (table test-table [id=ea0a909ca60f4ba6abe7c9d5078660aa]) successfully deleted
I20260621 05:19:48.843848 18979 ts_tablet_manager.cc:1918] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:48.843920 18979 log.cc:1196] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/85db7dca2b9746f5a0aaf6f6f3b8ef26
I20260621 05:19:48.844175 18979 ts_tablet_manager.cc:1939] T 85db7dca2b9746f5a0aaf6f6f3b8ef26 P 4ce75bac4035421eb21c3d77cb264bc4: Deleting consensus metadata
I20260621 05:19:48.844556 18468 catalog_manager.cc:4841] TS 4ce75bac4035421eb21c3d77cb264bc4 (127.0.98.3:40929): tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26 (table test-table [id=ea0a909ca60f4ba6abe7c9d5078660aa]) successfully deleted
W20260621 05:19:48.846406 18465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39359 (error 108)
I20260621 05:19:48.846628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 18677
I20260621 05:19:48.847282 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.846764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39359 (local address 127.0.0.1:60074)
0621 05:19:48.847198 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:48.847242 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
W20260621 05:19:48.847395 18465 catalog_manager.cc:4584] TS 09b4c509fc744cea891e20751314e66b (127.0.98.1:39359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85db7dca2b9746f5a0aaf6f6f3b8ef26: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39359: connect: Connection refused (error 111)
I20260621 05:19:48.851854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 18811
I20260621 05:19:48.856297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 18449
2026-06-21T05:19:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1822 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T05:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:48Z Disabled control of system clock
I20260621 05:19:48.883132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35555
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:19:48.962006 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:48.962216 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:48.962236 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:48.963253 18984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:48.963285 18984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:48.963297 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:48.963310 18984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:48.963320 18984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:48.964545 18984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:48.964728 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:48.964999 18984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:48.969684 18989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:48.969709 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:48.969820 18984 server_base.cc:1034] running on GCE node
W20260621 05:19:48.969736 18992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:48.970175 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:48.970499 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:48.971689 18984 hybrid_clock.cc:648] HybridClock initialized: now 1782019188971678 us; error 49 us; skew 500 ppm
I20260621 05:19:48.972872 18984 webserver.cc:460] Webserver started at http://127.0.98.62:40055/ using document root <none> and password file <none>
I20260621 05:19:48.973120 18984 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:48.973187 18984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:48.973315 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:48.974325 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "3aa9ffe151b640069859658631b242d3"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "245c3ec1cea4d45d52fa494048aedb1e"
server_key_iv: "9dd0a731c5122507362f924daf862af6"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.974680 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "3aa9ffe151b640069859658631b242d3"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "245c3ec1cea4d45d52fa494048aedb1e"
server_key_iv: "9dd0a731c5122507362f924daf862af6"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.975946 18984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:19:48.976823 18998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:48.976995 18984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:19:48.977093 18984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "3aa9ffe151b640069859658631b242d3"
format_stamp: "Formatted at 2026-06-21 05:19:48 on dist-test-slave-czm5"
server_key: "245c3ec1cea4d45d52fa494048aedb1e"
server_key_iv: "9dd0a731c5122507362f924daf862af6"
server_key_version: "encryptionkey@0"
I20260621 05:19:48.977187 18984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:48.988523 18984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:48.988818 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:48.988968 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:48.992970 18984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35555
I20260621 05:19:48.992995 19050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35555 every 8 connection(s)
I20260621 05:19:48.993302 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:48.994220 19051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:48.996387 19051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3: Bootstrap starting.
I20260621 05:19:48.997009 19051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:48.997182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 18984
I20260621 05:19:48.997298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:48.997350 19051 log.cc:824] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:48.997481 392 external_mini_cluster.cc:1428] Setting key 0e7614ebe48efe7778d0636a6284f134
I20260621 05:19:48.998174 19051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3: No bootstrap required, opened a new log
I20260621 05:19:49.000713 19051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9ffe151b640069859658631b242d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } }
I20260621 05:19:49.000874 19051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:49.000905 19051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa9ffe151b640069859658631b242d3, State: Initialized, Role: FOLLOWER
I20260621 05:19:49.001008 19051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9ffe151b640069859658631b242d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } }
I20260621 05:19:49.001075 19051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:49.001103 19051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:49.001149 19051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:49.001741 19051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9ffe151b640069859658631b242d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } }
I20260621 05:19:49.001852 19051 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aa9ffe151b640069859658631b242d3; no voters:
I20260621 05:19:49.002023 19051 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:49.002190 19057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:49.002313 19051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:49.002408 19057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3aa9ffe151b640069859658631b242d3, State: Running, Role: LEADER
I20260621 05:19:49.002494 19057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9ffe151b640069859658631b242d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } }
I20260621 05:19:49.002767 19057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa9ffe151b640069859658631b242d3. Latest consensus state: current_term: 1 leader_uuid: "3aa9ffe151b640069859658631b242d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9ffe151b640069859658631b242d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } } }
I20260621 05:19:49.002863 19057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:49.002780 19056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aa9ffe151b640069859658631b242d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9ffe151b640069859658631b242d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } } }
I20260621 05:19:49.002990 19056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:49.003254 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:48.997995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.0.98.62:35555)
0621 05:19:48.998191 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:48.998205 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:48.998261 (+ 56us) server_negotiation.cc:407] Connection header received
0621 05:19:48.998709 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:48.998715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:48.998795 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:48.998903 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:48.999146 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.000075 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.001199 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.001603 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.001688 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.002496 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:49.002500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:49.002501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:49.002705 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:49.002753 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:49.002768 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:49.002813 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.002964 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.003052 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
W20260621 05:19:49.004087 19071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:19:49.004154 19071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:19:49.004236 19064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:49.004433 19064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:49.005841 19064 catalog_manager.cc:1347] Generated new cluster ID: 5c241e6ad7c4425798c5dd5af173c047
I20260621 05:19:49.005880 19064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:49.011188 19064 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:49.011894 19064 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:49.018832 19064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3: Generated new TSK 0
I20260621 05:19:49.019007 19064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:19:49.021804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:19:49.122140 19075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:49.122365 19075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:49.122399 19075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:49.123495 19075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:49.123557 19075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:49.123598 19075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:19:49.124977 19075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:49.125234 19075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:49.125532 19075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:49.131327 19083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:49.131327 19081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:49.131346 19080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:49.131492 19075 server_base.cc:1034] running on GCE node
I20260621 05:19:49.131767 19075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:49.132030 19075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:49.133180 19075 hybrid_clock.cc:648] HybridClock initialized: now 1782019189133165 us; error 31 us; skew 500 ppm
I20260621 05:19:49.134403 19075 webserver.cc:460] Webserver started at http://127.0.98.1:46459/ using document root <none> and password file <none>
I20260621 05:19:49.134650 19075 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:49.134719 19075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:49.134871 19075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:49.135936 19075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "75b686380e80456aab25a161a9d39a3d"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "cc0d5071f953a6db9e13301559e9623c"
server_key_iv: "e0fe09f0ae45cdd132b7f6f7ed5fbbbb"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.136269 19075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "75b686380e80456aab25a161a9d39a3d"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "cc0d5071f953a6db9e13301559e9623c"
server_key_iv: "e0fe09f0ae45cdd132b7f6f7ed5fbbbb"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.137548 19075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:49.138322 19089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.138577 19075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:49.138674 19075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "75b686380e80456aab25a161a9d39a3d"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "cc0d5071f953a6db9e13301559e9623c"
server_key_iv: "e0fe09f0ae45cdd132b7f6f7ed5fbbbb"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.138749 19075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:49.152355 19075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:49.152663 19075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:49.152793 19075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:49.153014 19075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:49.153359 19075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:49.153402 19075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.153430 19075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:49.153456 19075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.160866 19075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38131
I20260621 05:19:49.160949 19202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38131 every 8 connection(s)
I20260621 05:19:49.161284 19075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:19:49.166630 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.162011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58411 (local address 127.0.98.62:35555)
0621 05:19:49.162053 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.162055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.162315 (+ 260us) server_negotiation.cc:407] Connection header received
0621 05:19:49.162842 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.162843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.162854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.162898 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:49.163544 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.164139 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.165151 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.165360 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.165431 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.166317 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:49.166319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:49.166319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:49.166324 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:49.166337 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:49.166341 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:49.166427 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.166527 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.166551 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:19:49.166841 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.162007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.1:58411)
0621 05:19:49.162188 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.162222 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:49.162738 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:49.162969 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:49.162972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:49.162980 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:49.163452 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.163457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.164235 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.164237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.165063 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.165065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.165449 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.165450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.165576 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.166151 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:49.166151 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:49.166194 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:49.166215 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:49.166427 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:49.166433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:49.166436 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:49.166516 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:19:49.166615 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260621 05:19:49.167285 19203 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35555
I20260621 05:19:49.167387 19203 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:49.167568 19203 heartbeater.cc:511] Master 127.0.98.62:35555 requested a full tablet report, sending...
I20260621 05:19:49.167681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 19075
I20260621 05:19:49.167781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:19:49.167995 392 external_mini_cluster.cc:1428] Setting key e6277a5bd3798cf1b4391a3f73c34816
I20260621 05:19:49.168226 19015 ts_manager.cc:194] Registered new tserver with Master: 75b686380e80456aab25a161a9d39a3d (127.0.98.1:38131)
I20260621 05:19:49.169070 19015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58411
I20260621 05:19:49.169435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:19:49.249382 19206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:49.249593 19206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:49.249614 19206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:49.250764 19206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:49.250804 19206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:49.250828 19206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:19:49.252138 19206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:49.252394 19206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:49.252700 19206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:49.257725 19211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:49.257716 19212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:49.257831 19206 server_base.cc:1034] running on GCE node
W20260621 05:19:49.257717 19214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:49.258311 19206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:49.258584 19206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:49.259749 19206 hybrid_clock.cc:648] HybridClock initialized: now 1782019189259734 us; error 40 us; skew 500 ppm
I20260621 05:19:49.260950 19206 webserver.cc:460] Webserver started at http://127.0.98.2:40163/ using document root <none> and password file <none>
I20260621 05:19:49.261196 19206 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:49.261263 19206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:49.261397 19206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:49.262475 19206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "725a8245851d477e889e1bba84affb9f"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "b60e88de7baa17303e839a1665cfb716"
server_key_iv: "2a5976063b7e432ba1b7f53053657a67"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.262842 19206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "725a8245851d477e889e1bba84affb9f"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "b60e88de7baa17303e839a1665cfb716"
server_key_iv: "2a5976063b7e432ba1b7f53053657a67"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.264232 19206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:19:49.264967 19220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.265148 19206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:49.265245 19206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "725a8245851d477e889e1bba84affb9f"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "b60e88de7baa17303e839a1665cfb716"
server_key_iv: "2a5976063b7e432ba1b7f53053657a67"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.265331 19206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:49.272832 19206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:49.273118 19206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:49.273262 19206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:49.273474 19206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:49.273782 19206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:49.273844 19206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.273885 19206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:49.273913 19206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.280885 19206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36069
I20260621 05:19:49.280925 19333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36069 every 8 connection(s)
I20260621 05:19:49.281260 19206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:19:49.284132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 19206
I20260621 05:19:49.284238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:19:49.284422 392 external_mini_cluster.cc:1428] Setting key 9c24a2f451803d1a14a9b03c4fe59d3c
I20260621 05:19:49.285377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:19:49.286985 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.282013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38171 (local address 127.0.98.62:35555)
0621 05:19:49.282058 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.282060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.282323 (+ 263us) server_negotiation.cc:407] Connection header received
0621 05:19:49.282794 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.282795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.282805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.282858 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:49.283493 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.284059 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.285487 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.285742 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.285804 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.286701 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:49.286703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:49.286703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:49.286708 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:49.286724 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:49.286728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:49.286753 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.286884 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.286910 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:19:49.287195 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.282018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.2:38171)
0621 05:19:49.282199 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.282232 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:49.282707 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:49.282925 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:49.282927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:49.282936 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:49.283424 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.283428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.284146 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.284147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.285381 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.285384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.285827 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.285828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.285988 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.286506 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:49.286506 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:49.286547 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:49.286579 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:49.286798 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:49.286802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:49.286804 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:19:49.286873 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:19:49.286970 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260621 05:19:49.287730 19334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35555
I20260621 05:19:49.287837 19334 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:49.288020 19334 heartbeater.cc:511] Master 127.0.98.62:35555 requested a full tablet report, sending...
I20260621 05:19:49.288409 19015 ts_manager.cc:194] Registered new tserver with Master: 725a8245851d477e889e1bba84affb9f (127.0.98.2:36069)
I20260621 05:19:49.288920 19015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38171
W20260621 05:19:49.367385 19337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:49.367722 19337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:49.367763 19337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:49.368984 19337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:49.369050 19337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:49.369091 19337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:19:49.370426 19337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:49.370608 19337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:49.370843 19337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:49.375787 19343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:49.375814 19345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:49.375787 19342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:49.375982 19337 server_base.cc:1034] running on GCE node
I20260621 05:19:49.376187 19337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:49.376466 19337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:49.377621 19337 hybrid_clock.cc:648] HybridClock initialized: now 1782019189377610 us; error 42 us; skew 500 ppm
I20260621 05:19:49.378765 19337 webserver.cc:460] Webserver started at http://127.0.98.3:43859/ using document root <none> and password file <none>
I20260621 05:19:49.379016 19337 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:49.379086 19337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:49.379264 19337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:49.380380 19337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "810e7988c9034a82a706aab41d7f3162"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "9b6d53c716befc078204d226167359e1"
server_key_iv: "2a8da276da4cec4014a3c3f749996248"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.380738 19337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "810e7988c9034a82a706aab41d7f3162"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "9b6d53c716befc078204d226167359e1"
server_key_iv: "2a8da276da4cec4014a3c3f749996248"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.381925 19337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:19:49.382586 19351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.382752 19337 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:49.382845 19337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "810e7988c9034a82a706aab41d7f3162"
format_stamp: "Formatted at 2026-06-21 05:19:49 on dist-test-slave-czm5"
server_key: "9b6d53c716befc078204d226167359e1"
server_key_iv: "2a8da276da4cec4014a3c3f749996248"
server_key_version: "encryptionkey@0"
I20260621 05:19:49.382944 19337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:49.396488 19337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:49.396795 19337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:49.396937 19337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:49.397147 19337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:49.397475 19337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:49.397536 19337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.397578 19337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:49.397605 19337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:49.403702 19337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36873
I20260621 05:19:49.403784 19464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36873 every 8 connection(s)
I20260621 05:19:49.404102 19337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:19:49.408895 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.404817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48989 (local address 127.0.98.62:35555)
0621 05:19:49.404905 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.404907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.405104 (+ 197us) server_negotiation.cc:407] Connection header received
0621 05:19:49.405566 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.405567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.405576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.405628 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:49.406233 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.406746 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.407765 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.407935 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.407986 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.408674 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:49.408675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:49.408675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:49.408679 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:49.408691 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:49.408695 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:49.408736 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.408817 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.408838 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 05:19:49.409085 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.404781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.3:48989)
0621 05:19:49.404984 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.405016 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:49.405483 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:49.405683 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:49.405686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:49.405693 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:49.406164 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.406168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.406822 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.406822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.407587 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.407589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.408004 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.408005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.408111 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.408551 (+ 440us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:49.408551 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:49.408577 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:49.408598 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:49.408747 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:49.408751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:49.408753 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:19:49.408802 (+ 49us) client_negotiation.cc:240] Negotiation successful
0621 05:19:49.408873 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260621 05:19:49.409646 19465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35555
I20260621 05:19:49.409736 19465 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:49.409927 19465 heartbeater.cc:511] Master 127.0.98.62:35555 requested a full tablet report, sending...
I20260621 05:19:49.410283 19015 ts_manager.cc:194] Registered new tserver with Master: 810e7988c9034a82a706aab41d7f3162 (127.0.98.3:36873)
I20260621 05:19:49.410744 19015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48989
I20260621 05:19:49.410799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 19337
I20260621 05:19:49.410894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:19:49.411094 392 external_mini_cluster.cc:1428] Setting key b14779ed3c94d62da82ef80c3c5973cb
I20260621 05:19:49.411906 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:19:49.415879 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.413237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.0.98.62:35555)
0621 05:19:49.413276 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.413278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.413379 (+ 101us) server_negotiation.cc:407] Connection header received
0621 05:19:49.413434 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.413435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.413443 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.413487 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:49.413626 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.414054 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.414712 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.414877 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.414923 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.415514 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:49.415515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:49.415516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:49.415519 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:49.415530 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:49.415533 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:49.415601 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.415743 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.415767 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:19:49.417320 19015 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:19:49.418963 19015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:19:49.425266 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.421382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.0.98.2:36069)
0621 05:19:49.421549 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.421554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.421567 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:19:49.421611 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.421613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.421629 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.421751 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:49.421924 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.422693 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.424066 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.424855 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.424929 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.425016 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.425127 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.425170 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260621 05:19:49.425271 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.421046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36069 (local address 127.0.0.1:34268)
0621 05:19:49.421237 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.421277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:49.421347 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:49.421753 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:49.421755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:49.421757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:49.421843 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.421845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.422790 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.422791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.423996 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.423998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.424946 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.424948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.425022 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.425033 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 05:19:49.425111 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:19:49.425179 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":52,"threads_started":1}
I20260621 05:19:49.425714 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.421096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38131 (local address 127.0.0.1:34084)
0621 05:19:49.421509 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.421518 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:49.421559 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:49.421844 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:49.421844 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:49.421846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:49.421891 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.421893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.423185 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.423186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.424314 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.424317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.425235 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.425236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.425299 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.425304 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:19:49.425335 (+ 31us) client_negotiation.cc:240] Negotiation successful
0621 05:19:49.425383 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":30,"thread_start_us":91,"threads_started":1}
I20260621 05:19:49.425727 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.421346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.98.3:36873)
0621 05:19:49.421489 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.421494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.421660 (+ 166us) server_negotiation.cc:407] Connection header received
0621 05:19:49.421710 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.421712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.421727 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.421804 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:49.421980 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.423089 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.424309 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.424872 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.424963 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.425050 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.425315 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.425629 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260621 05:19:49.425730 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.421191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.0.98.1:38131)
0621 05:19:49.421687 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.421690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.421699 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:49.421734 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.421736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.421753 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.421963 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:49.422027 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.423089 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.424398 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.425141 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.425484 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.425572 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.425604 (+ 32us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.425636 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":96,"threads_started":1}
I20260621 05:19:49.425278 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.421246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36873 (local address 127.0.0.1:34956)
0621 05:19:49.421575 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.421585 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:49.421648 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:49.421872 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:49.421873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:49.421874 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:49.421906 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.421907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.423185 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.423186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.424213 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.424216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.425083 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.425084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.425138 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.425141 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:49.425187 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:19:49.425215 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":36,"thread_start_us":68,"threads_started":1}
I20260621 05:19:49.426134 19264 tablet_service.cc:1467] Processing CreateTablet for tablet c7efdd68af0f49aaaa114ccf3c53dd00 (DEFAULT_TABLE table=test-table [id=1b13027ff9424aecbedfcc0ac9cc69f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:49.426280 19137 tablet_service.cc:1467] Processing CreateTablet for tablet c7efdd68af0f49aaaa114ccf3c53dd00 (DEFAULT_TABLE table=test-table [id=1b13027ff9424aecbedfcc0ac9cc69f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:49.426404 19399 tablet_service.cc:1467] Processing CreateTablet for tablet c7efdd68af0f49aaaa114ccf3c53dd00 (DEFAULT_TABLE table=test-table [id=1b13027ff9424aecbedfcc0ac9cc69f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:49.426508 19264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7efdd68af0f49aaaa114ccf3c53dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:49.426542 19137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7efdd68af0f49aaaa114ccf3c53dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:49.426636 19399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7efdd68af0f49aaaa114ccf3c53dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:49.429374 19479 tablet_bootstrap.cc:492] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Bootstrap starting.
I20260621 05:19:49.430016 19480 tablet_bootstrap.cc:492] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d: Bootstrap starting.
I20260621 05:19:49.430094 19479 tablet_bootstrap.cc:654] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:49.430377 19479 log.cc:824] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:49.430624 19480 tablet_bootstrap.cc:654] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:49.430723 19481 tablet_bootstrap.cc:492] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Bootstrap starting.
I20260621 05:19:49.430950 19480 log.cc:824] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:49.431126 19479 tablet_bootstrap.cc:492] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: No bootstrap required, opened a new log
I20260621 05:19:49.431188 19479 ts_tablet_manager.cc:1397] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:19:49.431362 19481 tablet_bootstrap.cc:654] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:49.431732 19481 log.cc:824] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:49.431833 19480 tablet_bootstrap.cc:492] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d: No bootstrap required, opened a new log
I20260621 05:19:49.431912 19480 ts_tablet_manager.cc:1397] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:19:49.432442 19481 tablet_bootstrap.cc:492] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: No bootstrap required, opened a new log
I20260621 05:19:49.432507 19481 ts_tablet_manager.cc:1397] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:19:49.433861 19479 raft_consensus.cc:348] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.434017 19479 raft_consensus.cc:374] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:49.434063 19479 raft_consensus.cc:729] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810e7988c9034a82a706aab41d7f3162, State: Initialized, Role: FOLLOWER
I20260621 05:19:49.434192 19479 consensus_queue.cc:260] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.434471 19479 ts_tablet_manager.cc:1428] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260621 05:19:49.434361 19481 raft_consensus.cc:348] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.434527 19481 raft_consensus.cc:374] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:49.434433 19480 raft_consensus.cc:348] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.434576 19480 raft_consensus.cc:374] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:49.434574 19481 raft_consensus.cc:729] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725a8245851d477e889e1bba84affb9f, State: Initialized, Role: FOLLOWER
I20260621 05:19:49.434619 19480 raft_consensus.cc:729] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b686380e80456aab25a161a9d39a3d, State: Initialized, Role: FOLLOWER
I20260621 05:19:49.434697 19465 heartbeater.cc:503] Master 127.0.98.62:35555 was elected leader, sending a full tablet report...
I20260621 05:19:49.434702 19481 consensus_queue.cc:260] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.434728 19480 consensus_queue.cc:260] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.434965 19480 ts_tablet_manager.cc:1428] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:19:49.434966 19481 ts_tablet_manager.cc:1428] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:19:49.435001 19334 heartbeater.cc:503] Master 127.0.98.62:35555 was elected leader, sending a full tablet report...
I20260621 05:19:49.435107 19203 heartbeater.cc:503] Master 127.0.98.62:35555 was elected leader, sending a full tablet report...
W20260621 05:19:49.531855 19335 tablet.cc:2369] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:19:49.654735 19466 tablet.cc:2369] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:19:49.662060 19204 tablet.cc:2369] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:19:49.707248 19485 raft_consensus.cc:482] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:49.707377 19485 raft_consensus.cc:504] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.707702 19485 leader_election.cc:288] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 725a8245851d477e889e1bba84affb9f (127.0.98.2:36069), 75b686380e80456aab25a161a9d39a3d (127.0.98.1:38131)
I20260621 05:19:49.711577 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.707992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50331 (local address 127.0.98.1:38131)
0621 05:19:49.708039 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.708041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.708051 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:19:49.708084 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.708085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.708096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.708158 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:49.708369 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.709141 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.710646 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.711199 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.711273 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.711283 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.711467 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.711504 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:19:49.711577 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.708014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36837 (local address 127.0.98.2:36069)
0621 05:19:49.708059 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.708061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.708214 (+ 153us) server_negotiation.cc:407] Connection header received
0621 05:19:49.708264 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.708265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.708273 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.708312 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:49.708472 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.709149 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.710646 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.711199 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.711273 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.711283 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.711467 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.711509 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:19:49.711594 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.707885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38131 (local address 127.0.98.3:50331)
0621 05:19:49.707956 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.707969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:49.708026 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:49.708188 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:49.708189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:49.708191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:49.708264 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.708266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.709228 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.709229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.710550 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.710553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.711295 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.711297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.711378 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.711383 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:19:49.711445 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:19:49.711497 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":53}
I20260621 05:19:49.711603 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.707885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36069 (local address 127.0.98.3:36837)
0621 05:19:49.708131 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.708157 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:49.708207 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:49.708335 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:49.708335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:49.708348 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:49.708406 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.708407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.709228 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.709229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.710550 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:49.710553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.711295 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.711297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:49.711387 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.711389 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:19:49.711445 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:19:49.711496 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":161,"thread_start_us":96,"threads_started":1}
I20260621 05:19:49.712006 19157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7efdd68af0f49aaaa114ccf3c53dd00" candidate_uuid: "810e7988c9034a82a706aab41d7f3162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b686380e80456aab25a161a9d39a3d" is_pre_election: true
I20260621 05:19:49.712191 19157 raft_consensus.cc:2454] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 810e7988c9034a82a706aab41d7f3162 in term 0.
I20260621 05:19:49.712430 19288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7efdd68af0f49aaaa114ccf3c53dd00" candidate_uuid: "810e7988c9034a82a706aab41d7f3162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725a8245851d477e889e1bba84affb9f" is_pre_election: true
I20260621 05:19:49.712509 19355 leader_election.cc:302] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75b686380e80456aab25a161a9d39a3d, 810e7988c9034a82a706aab41d7f3162; no voters:
I20260621 05:19:49.712596 19288 raft_consensus.cc:2454] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 810e7988c9034a82a706aab41d7f3162 in term 0.
I20260621 05:19:49.712644 19485 raft_consensus.cc:2781] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:19:49.712693 19485 raft_consensus.cc:482] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:19:49.712714 19485 raft_consensus.cc:3037] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:49.713435 19485 raft_consensus.cc:504] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.713585 19485 leader_election.cc:288] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [CANDIDATE]: Term 1 election: Requested vote from peers 725a8245851d477e889e1bba84affb9f (127.0.98.2:36069), 75b686380e80456aab25a161a9d39a3d (127.0.98.1:38131)
I20260621 05:19:49.713730 19157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7efdd68af0f49aaaa114ccf3c53dd00" candidate_uuid: "810e7988c9034a82a706aab41d7f3162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b686380e80456aab25a161a9d39a3d"
I20260621 05:19:49.713734 19288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7efdd68af0f49aaaa114ccf3c53dd00" candidate_uuid: "810e7988c9034a82a706aab41d7f3162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725a8245851d477e889e1bba84affb9f"
I20260621 05:19:49.713811 19157 raft_consensus.cc:3037] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:49.713811 19288 raft_consensus.cc:3037] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:49.714676 19157 raft_consensus.cc:2454] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 810e7988c9034a82a706aab41d7f3162 in term 1.
I20260621 05:19:49.714676 19288 raft_consensus.cc:2454] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 810e7988c9034a82a706aab41d7f3162 in term 1.
I20260621 05:19:49.714854 19353 leader_election.cc:302] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 725a8245851d477e889e1bba84affb9f, 810e7988c9034a82a706aab41d7f3162; no voters:
I20260621 05:19:49.714964 19485 raft_consensus.cc:2781] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:49.715131 19485 raft_consensus.cc:686] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 1 LEADER]: Becoming Leader. State: Replica: 810e7988c9034a82a706aab41d7f3162, State: Running, Role: LEADER
I20260621 05:19:49.715235 19485 consensus_queue.cc:237] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260621 05:19:49.716063 19014 catalog_manager.cc:5495] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 810e7988c9034a82a706aab41d7f3162 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "810e7988c9034a82a706aab41d7f3162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "810e7988c9034a82a706aab41d7f3162" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20260621 05:19:49.781263 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.778529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.0.98.3:36873)
0621 05:19:49.778591 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:49.778594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:49.778603 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:49.778676 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:49.778677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:49.778697 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:49.778781 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:19:49.778927 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.779440 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.780404 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:49.780597 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:49.780689 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:49.780882 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:19:49.780991 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:19:49.781103 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:49.781154 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 05:19:49.781163 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:19:49.783149 19288 raft_consensus.cc:1261] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 1 FOLLOWER]: Refusing update from remote peer 810e7988c9034a82a706aab41d7f3162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:49.783147 19157 raft_consensus.cc:1261] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 1 FOLLOWER]: Refusing update from remote peer 810e7988c9034a82a706aab41d7f3162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:49.783541 19489 consensus_queue.cc:1035] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "725a8245851d477e889e1bba84affb9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:49.783706 19485 consensus_queue.cc:1035] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b686380e80456aab25a161a9d39a3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:49.786007 19492 mvcc.cc:205] Tried to move back new op lower bound from 7299150601348849664 to 7299150601074196480. Current Snapshot: MvccSnapshot[applied={T|T < 7299150601348849664}]
I20260621 05:19:49.786250 19493 mvcc.cc:205] Tried to move back new op lower bound from 7299150601348849664 to 7299150601074196480. Current Snapshot: MvccSnapshot[applied={T|T < 7299150601348849664}]
I20260621 05:19:49.786521 19494 mvcc.cc:205] Tried to move back new op lower bound from 7299150601348849664 to 7299150601074196480. Current Snapshot: MvccSnapshot[applied={T|T < 7299150601348849664}]
I20260621 05:19:49.789137 19014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:49.789243 19014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:49.790103 19014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aa9ffe151b640069859658631b242d3: Sending DeleteTablet for 3 replicas of tablet c7efdd68af0f49aaaa114ccf3c53dd00
I20260621 05:19:49.790445 19137 tablet_service.cc:1514] Processing DeleteTablet for tablet c7efdd68af0f49aaaa114ccf3c53dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:49 UTC) from {username='slave'} at 127.0.0.1:34084
I20260621 05:19:49.790485 19264 tablet_service.cc:1514] Processing DeleteTablet for tablet c7efdd68af0f49aaaa114ccf3c53dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:49 UTC) from {username='slave'} at 127.0.0.1:34268
I20260621 05:19:49.790512 19399 tablet_service.cc:1514] Processing DeleteTablet for tablet c7efdd68af0f49aaaa114ccf3c53dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:49 UTC) from {username='slave'} at 127.0.0.1:34956
I20260621 05:19:49.790737 19503 tablet_replica.cc:330] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: stopping tablet replica
I20260621 05:19:49.790817 19504 tablet_replica.cc:330] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: stopping tablet replica
I20260621 05:19:49.790917 19504 raft_consensus.cc:2229] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:19:49.791036 19502 tablet_replica.cc:330] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d: stopping tablet replica
I20260621 05:19:49.791047 19504 raft_consensus.cc:2258] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:49.791134 19502 raft_consensus.cc:2229] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:49.791149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 19075
I20260621 05:19:49.791258 19502 raft_consensus.cc:2258] T c7efdd68af0f49aaaa114ccf3c53dd00 P 75b686380e80456aab25a161a9d39a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:49.791505 19504 ts_tablet_manager.cc:1905] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:49.793538 19504 ts_tablet_manager.cc:1918] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:49.793609 19504 log.cc:1196] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/c7efdd68af0f49aaaa114ccf3c53dd00
I20260621 05:19:49.793871 19504 ts_tablet_manager.cc:1939] T c7efdd68af0f49aaaa114ccf3c53dd00 P 810e7988c9034a82a706aab41d7f3162: Deleting consensus metadata
I20260621 05:19:49.794364 19000 catalog_manager.cc:4841] TS 810e7988c9034a82a706aab41d7f3162 (127.0.98.3:36873): tablet c7efdd68af0f49aaaa114ccf3c53dd00 (table test-table [id=1b13027ff9424aecbedfcc0ac9cc69f2]) successfully deleted
I20260621 05:19:49.790937 19503 raft_consensus.cc:2229] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:49.795871 19503 raft_consensus.cc:2258] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:49.796370 19503 ts_tablet_manager.cc:1905] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:49.798382 19503 ts_tablet_manager.cc:1918] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:49.798483 19503 log.cc:1196] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/c7efdd68af0f49aaaa114ccf3c53dd00
I20260621 05:19:49.798768 19503 ts_tablet_manager.cc:1939] T c7efdd68af0f49aaaa114ccf3c53dd00 P 725a8245851d477e889e1bba84affb9f: Deleting consensus metadata
W20260621 05:19:49.798923 19002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38131 (error 108)
I20260621 05:19:49.799188 19000 catalog_manager.cc:4841] TS 725a8245851d477e889e1bba84affb9f (127.0.98.2:36069): tablet c7efdd68af0f49aaaa114ccf3c53dd00 (table test-table [id=1b13027ff9424aecbedfcc0ac9cc69f2]) successfully deleted
I20260621 05:19:49.799402 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:49.799277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38131 (local address 127.0.0.1:34094)
0621 05:19:49.799322 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:49.799363 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260621 05:19:49.799497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 19206
W20260621 05:19:49.799547 19002 catalog_manager.cc:4584] TS 75b686380e80456aab25a161a9d39a3d (127.0.98.1:38131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7efdd68af0f49aaaa114ccf3c53dd00: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38131: connect: Connection refused (error 111)
I20260621 05:19:49.805229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 19337
I20260621 05:19:49.809825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 18984
2026-06-21T05:19:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (953 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:49 dist-test-slave-czm5 krb5kdc[19508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:49 dist-test-slave-czm5 krb5kdc[19508](info): set up 2 sockets
Jun 21 05:19:49 dist-test-slave-czm5 krb5kdc[19508](info): commencing operation
krb5kdc: starting...
W20260621 05:19:50.843237 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:19:50 dist-test-slave-czm5 krb5kdc[19508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:19:50.953794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44403
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:51.032804 19521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:51.033046 19521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:51.033082 19521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:51.034183 19521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:51.034257 19521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:51.034286 19521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:51.034317 19521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:51.034339 19521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:51.035764 19521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:51.036026 19521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:51.036248 19521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:51.041047 19529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:51.041040 19526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:51.041039 19527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:51.041124 19521 server_base.cc:1034] running on GCE node
I20260621 05:19:51.041471 19521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:51.041767 19521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:51.042935 19521 hybrid_clock.cc:648] HybridClock initialized: now 1782019191042920 us; error 27 us; skew 500 ppm
Jun 21 05:19:51 dist-test-slave-czm5 krb5kdc[19508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:51.045320 19521 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:19:51.045727 19521 webserver.cc:460] Webserver started at http://127.0.98.62:36161/ using document root <none> and password file <none>
I20260621 05:19:51.045976 19521 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:51.046038 19521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:51.046172 19521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:51.047158 19521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "3ed5c07033bf419b86a2f22026a47e65"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "d1bc2d4ef1f05d4f583ebbf6976f179f"
server_key_iv: "a48fec3aaeec136a055293e50932fd37"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.047513 19521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ed5c07033bf419b86a2f22026a47e65"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "d1bc2d4ef1f05d4f583ebbf6976f179f"
server_key_iv: "a48fec3aaeec136a055293e50932fd37"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.048820 19521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:51.049603 19536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.049831 19521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:51.049923 19521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "3ed5c07033bf419b86a2f22026a47e65"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "d1bc2d4ef1f05d4f583ebbf6976f179f"
server_key_iv: "a48fec3aaeec136a055293e50932fd37"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.049976 19521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:51.056362 19521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:51.058279 19521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:51.058430 19521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:51.062274 19521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44403
I20260621 05:19:51.062369 19588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44403 every 8 connection(s)
I20260621 05:19:51.062693 19521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:51.063519 19589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:51.065528 19589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65: Bootstrap starting.
I20260621 05:19:51.066043 19589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:51.066308 19589 log.cc:824] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:51.066963 19589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65: No bootstrap required, opened a new log
I20260621 05:19:51.068650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 19521
I20260621 05:19:51.068584 19589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed5c07033bf419b86a2f22026a47e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } }
I20260621 05:19:51.068750 19589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:51.068769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:51.068797 19589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed5c07033bf419b86a2f22026a47e65, State: Initialized, Role: FOLLOWER
I20260621 05:19:51.068960 392 external_mini_cluster.cc:1428] Setting key fb960764dbda7765721491dcbd453db5
I20260621 05:19:51.068954 19589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed5c07033bf419b86a2f22026a47e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } }
I20260621 05:19:51.069031 19589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:51.069074 19589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:51.069474 19589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:51.070602 19589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed5c07033bf419b86a2f22026a47e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } }
I20260621 05:19:51.070720 19589 leader_election.cc:302] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ed5c07033bf419b86a2f22026a47e65; no voters:
I20260621 05:19:51.070947 19589 leader_election.cc:288] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:51.070972 19594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:51.071079 19594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [term 1 LEADER]: Becoming Leader. State: Replica: 3ed5c07033bf419b86a2f22026a47e65, State: Running, Role: LEADER
I20260621 05:19:51.071157 19594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed5c07033bf419b86a2f22026a47e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } }
I20260621 05:19:51.071193 19589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:51.071611 19596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ed5c07033bf419b86a2f22026a47e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed5c07033bf419b86a2f22026a47e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } }
I20260621 05:19:51.071741 19596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:51.071609 19594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ed5c07033bf419b86a2f22026a47e65. Latest consensus state: current_term: 1 leader_uuid: "3ed5c07033bf419b86a2f22026a47e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed5c07033bf419b86a2f22026a47e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } }
I20260621 05:19:51.071995 19594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:51.072245 19600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:51.072765 19600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:51.075598 19600 catalog_manager.cc:1347] Generated new cluster ID: 6fad3d0e6708428f95345178ba467d4c
I20260621 05:19:51.075667 19600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:19:51 dist-test-slave-czm5 krb5kdc[19508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:51.079775 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.069395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.98.62:44403)
0621 05:19:51.069566 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.069579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.069630 (+ 51us) server_negotiation.cc:407] Connection header received
0621 05:19:51.069745 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.069751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.069799 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.069908 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:51.070779 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.072082 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.073062 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.073439 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.073515 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.076183 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:51.076187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:51.076188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:51.076197 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:51.078052 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.078466 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.078468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.078469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.078573 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.078779 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.078780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.078780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.079061 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:51.079199 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.079329 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.079540 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260621 05:19:51.092012 19600 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:51.092491 19600 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:51.097167 19600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65: Generated new TSK 0
I20260621 05:19:51.097345 19600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260621 05:19:51.120633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:51.198766 19615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:51.198959 19615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:51.198993 19615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:51.200146 19615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:51.200186 19615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:51.200212 19615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:19:51.201628 19615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:51.201812 19615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:51.201989 19615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:51.206894 19621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:51.206890 19623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:51.206890 19620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:51.207098 19615 server_base.cc:1034] running on GCE node
I20260621 05:19:51.207300 19615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:51.207563 19615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:51.208702 19615 hybrid_clock.cc:648] HybridClock initialized: now 1782019191208694 us; error 29 us; skew 500 ppm
Jun 21 05:19:51 dist-test-slave-czm5 krb5kdc[19508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:51.211077 19615 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260621 05:19:51.211472 19615 webserver.cc:460] Webserver started at http://127.0.98.1:39265/ using document root <none> and password file <none>
I20260621 05:19:51.211733 19615 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:51.211798 19615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:51.211956 19615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:51.212940 19615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd26a06992144373b508bdb120fe2a34"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "aba6726d99b2c0ddf6ac7dec4bafede7"
server_key_iv: "12c53a1884a0445838cf8bd6f755fa8c"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.213284 19615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd26a06992144373b508bdb120fe2a34"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "aba6726d99b2c0ddf6ac7dec4bafede7"
server_key_iv: "12c53a1884a0445838cf8bd6f755fa8c"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.214613 19615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:19:51.215319 19630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.215505 19615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:51.215600 19615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "bd26a06992144373b508bdb120fe2a34"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "aba6726d99b2c0ddf6ac7dec4bafede7"
server_key_iv: "12c53a1884a0445838cf8bd6f755fa8c"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.215680 19615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:51.236166 19615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:51.238511 19615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:51.238677 19615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:51.238917 19615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:51.239234 19615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:51.239287 19615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.239320 19615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:51.239351 19615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.245015 19615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41267
I20260621 05:19:51.245070 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41267 every 8 connection(s)
I20260621 05:19:51.245421 19615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:19:51 dist-test-slave-czm5 krb5kdc[19508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:51.254451 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.246101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33619 (local address 127.0.98.62:44403)
0621 05:19:51.246169 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.246171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.246388 (+ 217us) server_negotiation.cc:407] Connection header received
0621 05:19:51.247272 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.247274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.247288 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.247342 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:51.247982 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.248476 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.249436 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.249636 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.249709 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.251864 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:51.251866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:51.251866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:51.251871 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:51.253317 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.253718 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.253718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.253719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.253767 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.253978 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.253979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.253979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.254075 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:51.254126 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.254333 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.254389 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:19:51.254638 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.246084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.1:33619)
0621 05:19:51.246257 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.246292 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:51.247173 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:51.247398 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:51.247401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:51.247698 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:51.247897 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.247902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.248553 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.248554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.249333 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.249334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.249723 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.249723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.249816 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.250307 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:51.250326 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:51.251787 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:51.253402 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:51.253408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:51.253409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:51.253651 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:51.253827 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:51.253827 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:51.253828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:51.253924 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:51.254130 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:51.254134 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:51.254238 (+ 104us) client_negotiation.cc:769] Sending connection context
0621 05:19:51.254314 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:19:51.254438 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260621 05:19:51.255121 19744 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260621 05:19:51.255218 19744 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:51.255300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 19615
I20260621 05:19:51.255416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:19:51.255429 19744 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260621 05:19:51.255636 392 external_mini_cluster.cc:1428] Setting key 818c5847b398eaf7dc8657c66185c7cd
I20260621 05:19:51.256031 19553 ts_manager.cc:194] Registered new tserver with Master: bd26a06992144373b508bdb120fe2a34 (127.0.98.1:41267)
I20260621 05:19:51.256738 19553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33619
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260621 05:19:51.278113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:51.358124 19749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:51.358346 19749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:51.358383 19749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:51.359464 19749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:51.359529 19749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:51.359573 19749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:19:51.361099 19749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:51.361337 19749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:51.361550 19749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:51.366284 19754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:51.366259 19755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:51.366403 19749 server_base.cc:1034] running on GCE node
W20260621 05:19:51.366267 19757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:51.366665 19749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:51.366947 19749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:51.368103 19749 hybrid_clock.cc:648] HybridClock initialized: now 1782019191368093 us; error 38 us; skew 500 ppm
Jun 21 05:19:51 dist-test-slave-czm5 krb5kdc[19508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:51.370486 19749 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260621 05:19:51.371012 19749 webserver.cc:460] Webserver started at http://127.0.98.2:43867/ using document root <none> and password file <none>
I20260621 05:19:51.371227 19749 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:51.371302 19749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:51.371466 19749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:51.372668 19749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "b42dd2daafa449868f37518e7fb54011"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "cd0046c53e20ac6fa3ad34b320a8c9bc"
server_key_iv: "800ae14ba60632fde82f9acc74635e01"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.373014 19749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b42dd2daafa449868f37518e7fb54011"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "cd0046c53e20ac6fa3ad34b320a8c9bc"
server_key_iv: "800ae14ba60632fde82f9acc74635e01"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.374271 19749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:19:51.374967 19764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.375129 19749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:51.375232 19749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "b42dd2daafa449868f37518e7fb54011"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "cd0046c53e20ac6fa3ad34b320a8c9bc"
server_key_iv: "800ae14ba60632fde82f9acc74635e01"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.375303 19749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:51.408834 19749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:51.411239 19749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:51.411422 19749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:51.411737 19749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:51.412055 19749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:51.412109 19749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.412150 19749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:51.412175 19749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.418215 19749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39617
I20260621 05:19:51.418277 19877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39617 every 8 connection(s)
I20260621 05:19:51.418665 19749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:19:51.423059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 19749
I20260621 05:19:51.423174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:19:51.423386 392 external_mini_cluster.cc:1428] Setting key e72a6cef140a864589871e990a82e396
Jun 21 05:19:51 dist-test-slave-czm5 krb5kdc[19508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:51.431972 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.419426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57271 (local address 127.0.98.62:44403)
0621 05:19:51.419496 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.419498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.419841 (+ 343us) server_negotiation.cc:407] Connection header received
0621 05:19:51.420992 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.420994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.421007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.421061 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:51.421972 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.423052 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.424878 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.425252 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.425352 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.428207 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:51.428209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:51.428209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:51.428216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:51.430049 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.430565 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.430566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.430566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.430629 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.430964 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.430966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.430966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.431255 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:51.431314 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.431748 (+ 434us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.431837 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:19:51.432099 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.419446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.2:57271)
0621 05:19:51.419603 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.419667 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:51.420873 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:51.421132 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:51.421136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:51.421570 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:51.421854 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.421861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.423262 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.423264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.424750 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.424753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.425377 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.425379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.425521 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.426086 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:51.426097 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:51.428091 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:51.430166 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:51.430173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:51.430175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:51.430479 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:51.430702 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:51.430703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:51.430703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:51.430849 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:51.431353 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:51.431360 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:51.431529 (+ 169us) client_negotiation.cc:769] Sending connection context
0621 05:19:51.431682 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 05:19:51.431858 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260621 05:19:51.433131 19878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260621 05:19:51.433251 19878 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:51.433462 19878 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260621 05:19:51.433871 19553 ts_manager.cc:194] Registered new tserver with Master: b42dd2daafa449868f37518e7fb54011 (127.0.98.2:39617)
I20260621 05:19:51.434530 19553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57271
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260621 05:19:51.452468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:51.531793 19883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:51.531993 19883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:51.532016 19883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:51.533134 19883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:51.533174 19883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:51.533197 19883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:19:51.534621 19883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:51.534791 19883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:51.535046 19883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:51.539873 19889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:51.539983 19883 server_base.cc:1034] running on GCE node
W20260621 05:19:51.539921 19891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:51.539925 19888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:51.540304 19883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:51.540541 19883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:51.541699 19883 hybrid_clock.cc:648] HybridClock initialized: now 1782019191541695 us; error 37 us; skew 500 ppm
Jun 21 05:19:51 dist-test-slave-czm5 krb5kdc[19508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:51.544253 19883 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260621 05:19:51.544665 19883 webserver.cc:460] Webserver started at http://127.0.98.3:41617/ using document root <none> and password file <none>
I20260621 05:19:51.544884 19883 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:51.544958 19883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:51.545110 19883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:51.546139 19883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "2f4307b5e7214d16a78cc89b90b3a2f0"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "45fce947c255428729002359eca12ebe"
server_key_iv: "c938fc081d9b5c35593d10b0b38d526c"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.546485 19883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2f4307b5e7214d16a78cc89b90b3a2f0"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "45fce947c255428729002359eca12ebe"
server_key_iv: "c938fc081d9b5c35593d10b0b38d526c"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.547784 19883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:19:51.548492 19898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.548667 19883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:51.548763 19883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "2f4307b5e7214d16a78cc89b90b3a2f0"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "45fce947c255428729002359eca12ebe"
server_key_iv: "c938fc081d9b5c35593d10b0b38d526c"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.548851 19883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:51.572407 19883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:51.574689 19883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:51.574849 19883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:51.575106 19883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:51.575392 19883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:51.575464 19883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.575510 19883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:51.575534 19883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.581655 19883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41975
I20260621 05:19:51.581754 20011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41975 every 8 connection(s)
I20260621 05:19:51.582042 19883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:19:51.587224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 19883
I20260621 05:19:51.587311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:19:51.587479 392 external_mini_cluster.cc:1428] Setting key 6fd6c36de87f68ad032a0973c68b0494
Jun 21 05:19:51 dist-test-slave-czm5 krb5kdc[19508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:19:51.591097 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.582741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60367 (local address 127.0.98.62:44403)
0621 05:19:51.582788 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.582790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.583039 (+ 249us) server_negotiation.cc:407] Connection header received
0621 05:19:51.584013 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.584014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.584024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.584076 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:51.584613 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.585120 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.586074 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.586248 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.586296 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.588601 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:51.588603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:51.588603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:51.588611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:51.589717 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.590116 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.590118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.590118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.590178 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.590450 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.590451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.590452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.590566 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:51.590643 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.590925 (+ 282us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.591014 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:19:51.591266 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.582746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.3:60367)
0621 05:19:51.582922 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.582956 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:51.583929 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:51.584126 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:51.584129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:51.584358 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:51.584547 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.584551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.585197 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.585197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.586005 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.586006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.586315 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.586316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.586409 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.586881 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:19:51.586889 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:51.588510 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:51.589814 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:51.589820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:51.589822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:51.590038 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:51.590264 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:19:51.590264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:19:51.590265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:19:51.590375 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:19:51.590635 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:51.590640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:51.590782 (+ 142us) client_negotiation.cc:769] Sending connection context
0621 05:19:51.590890 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 05:19:51.591044 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260621 05:19:51.591697 20012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260621 05:19:51.591778 20012 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:51.591933 20012 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260621 05:19:51.592271 19553 ts_manager.cc:194] Registered new tserver with Master: 2f4307b5e7214d16a78cc89b90b3a2f0 (127.0.98.3:41975)
I20260621 05:19:51.592625 19553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60367
I20260621 05:19:51.598974 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:19:51.606004 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.600266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.0.98.62:44403)
0621 05:19:51.600326 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.600329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.600436 (+ 107us) server_negotiation.cc:407] Connection header received
0621 05:19:51.600511 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.600511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.600521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.600589 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:51.601161 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.601620 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.602322 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.602498 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.602573 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.603798 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:51.603800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:51.603800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:51.603807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:51.604864 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.605196 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.605198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.605198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.605255 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:51.605463 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:51.605464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:51.605464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:51.605596 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:51.605733 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.605836 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.605909 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:19:51.607437 19553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:19:51.608964 19553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:19:51.615272 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.611242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.0.98.2:39617)
0621 05:19:51.611429 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.611434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.611448 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:19:51.611494 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.611496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.611510 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.611609 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:51.611820 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.612927 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.614204 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.614915 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.614994 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.615062 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.615145 (+ 83us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.615183 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260621 05:19:51.615288 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.611129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39617 (local address 127.0.0.1:55376)
0621 05:19:51.611373 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.611384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:51.611437 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:51.611649 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:51.611652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:51.611654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:51.611728 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.611730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.613039 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.613040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.614120 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.614122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.614998 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.614999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.615067 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.615077 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 05:19:51.615149 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:19:51.615198 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":45,"thread_start_us":75,"threads_started":1}
I20260621 05:19:51.615383 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.611297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41975 (local address 127.0.0.1:43954)
0621 05:19:51.611513 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.611522 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:51.611558 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:51.611911 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:51.611912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:51.611914 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:51.611952 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.611954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.613161 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.613162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.614314 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.614317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.615172 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.615173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.615220 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.615222 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:19:51.615271 (+ 49us) client_negotiation.cc:240] Negotiation successful
0621 05:19:51.615304 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":52,"thread_start_us":68,"threads_started":1}
I20260621 05:19:51.615708 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.611482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.0.98.3:41975)
0621 05:19:51.611726 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.611729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.611738 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:51.611788 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.611790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.611806 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.612022 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:51.612096 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.613073 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.614402 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.615098 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.615162 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.615209 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.615571 (+ 362us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.615611 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
I20260621 05:19:51.616142 19812 tablet_service.cc:1467] Processing CreateTablet for tablet 2e30713709e149e782e1d20f1a674cb2 (DEFAULT_TABLE table=test-table [id=80ce81936f5144a48399b69c9ecfcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:51.616261 19946 tablet_service.cc:1467] Processing CreateTablet for tablet 2e30713709e149e782e1d20f1a674cb2 (DEFAULT_TABLE table=test-table [id=80ce81936f5144a48399b69c9ecfcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:51.616515 19812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e30713709e149e782e1d20f1a674cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:51.616588 19946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e30713709e149e782e1d20f1a674cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:51.616799 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.611097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41267 (local address 127.0.0.1:47652)
0621 05:19:51.611267 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.611312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:51.611375 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:51.612121 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:51.612121 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:51.612123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:51.612182 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.612183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.613017 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.613018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.614314 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.614317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.616488 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.616489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.616589 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.616595 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:19:51.616652 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:19:51.616704 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":40,"thread_start_us":62,"threads_started":1}
I20260621 05:19:51.617074 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.611197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.0.98.1:41267)
0621 05:19:51.611786 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.611791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.611801 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:19:51.611899 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.611912 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.611958 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.612034 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:51.612257 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.612917 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.615245 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.616280 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.616357 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.616410 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.616946 (+ 536us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.616994 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":87,"threads_started":1}
I20260621 05:19:51.619292 20029 tablet_bootstrap.cc:492] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: Bootstrap starting.
I20260621 05:19:51.619315 20030 tablet_bootstrap.cc:492] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Bootstrap starting.
I20260621 05:19:51.619994 20029 tablet_bootstrap.cc:654] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:51.620074 20030 tablet_bootstrap.cc:654] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:51.620040 19678 tablet_service.cc:1467] Processing CreateTablet for tablet 2e30713709e149e782e1d20f1a674cb2 (DEFAULT_TABLE table=test-table [id=80ce81936f5144a48399b69c9ecfcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:51.620410 19678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e30713709e149e782e1d20f1a674cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:51.620554 20029 log.cc:824] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:51.620596 20030 log.cc:824] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:51.621294 20029 tablet_bootstrap.cc:492] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: No bootstrap required, opened a new log
I20260621 05:19:51.621374 20029 ts_tablet_manager.cc:1397] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:19:51.622967 20029 raft_consensus.cc:348] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
I20260621 05:19:51.623145 20029 raft_consensus.cc:374] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:51.623196 20029 raft_consensus.cc:729] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4307b5e7214d16a78cc89b90b3a2f0, State: Initialized, Role: FOLLOWER
I20260621 05:19:51.623224 20033 tablet_bootstrap.cc:492] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34: Bootstrap starting.
I20260621 05:19:51.623335 20029 consensus_queue.cc:260] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
I20260621 05:19:51.623651 20029 ts_tablet_manager.cc:1428] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:19:51.623670 20012 heartbeater.cc:503] Master 127.0.98.62:44403 was elected leader, sending a full tablet report...
I20260621 05:19:51.623865 20033 tablet_bootstrap.cc:654] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:51.624207 20030 tablet_bootstrap.cc:492] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: No bootstrap required, opened a new log
I20260621 05:19:51.624279 20030 ts_tablet_manager.cc:1397] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260621 05:19:51.624444 20033 log.cc:824] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:51.625113 20033 tablet_bootstrap.cc:492] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34: No bootstrap required, opened a new log
I20260621 05:19:51.625190 20033 ts_tablet_manager.cc:1397] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:19:51.625826 20030 raft_consensus.cc:348] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
I20260621 05:19:51.625950 20030 raft_consensus.cc:374] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:51.625972 20030 raft_consensus.cc:729] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42dd2daafa449868f37518e7fb54011, State: Initialized, Role: FOLLOWER
I20260621 05:19:51.626088 20030 consensus_queue.cc:260] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
I20260621 05:19:51.626355 20030 ts_tablet_manager.cc:1428] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:19:51.626369 19878 heartbeater.cc:503] Master 127.0.98.62:44403 was elected leader, sending a full tablet report...
I20260621 05:19:51.626879 20033 raft_consensus.cc:348] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
I20260621 05:19:51.627101 20033 raft_consensus.cc:374] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:51.627152 20033 raft_consensus.cc:729] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd26a06992144373b508bdb120fe2a34, State: Initialized, Role: FOLLOWER
I20260621 05:19:51.627278 20033 consensus_queue.cc:260] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
I20260621 05:19:51.627514 20033 ts_tablet_manager.cc:1428] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:19:51.627578 19744 heartbeater.cc:503] Master 127.0.98.62:44403 was elected leader, sending a full tablet report...
W20260621 05:19:51.669477 19879 tablet.cc:2369] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:19:51.745745 20037 raft_consensus.cc:482] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:51.745864 20037 raft_consensus.cc:504] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
W20260621 05:19:51.746223 19745 tablet.cc:2369] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:19:51.746240 20037 leader_election.cc:288] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f4307b5e7214d16a78cc89b90b3a2f0 (127.0.98.3:41975), b42dd2daafa449868f37518e7fb54011 (127.0.98.2:39617)
I20260621 05:19:51.749060 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.746532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45899 (local address 127.0.98.3:41975)
0621 05:19:51.746596 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.746598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.746607 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:51.746642 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.746643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.746650 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.746748 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:51.746901 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.747357 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.748352 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.748699 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.748793 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.748802 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.748960 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.748995 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:19:51.749091 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.746450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41975 (local address 127.0.98.1:45899)
0621 05:19:51.746512 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.746523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:51.746567 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:51.746740 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:51.746741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:51.746743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:51.746827 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.746828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.747439 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.747440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.748250 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.748252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.748792 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.748793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.748875 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.748879 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:51.748952 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:19:51.748990 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 05:19:51.749188 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.746450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39617 (local address 127.0.98.1:40715)
0621 05:19:51.746664 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.746670 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:51.746730 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:51.746875 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:51.746876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:51.746877 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:51.746951 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.746953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.747547 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.747548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.748314 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:51.748317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.748839 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.748840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:51.748918 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.748921 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:51.748994 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:19:51.749032 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":83,"thread_start_us":81,"threads_started":1}
I20260621 05:19:51.749102 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.746541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40715 (local address 127.0.98.2:39617)
0621 05:19:51.746582 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.746585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.746728 (+ 143us) server_negotiation.cc:407] Connection header received
0621 05:19:51.746783 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.746785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.746797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.746872 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:51.747027 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.747466 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.748401 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.748755 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.748837 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.748846 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.749002 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.749034 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:19:51.749481 19966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e30713709e149e782e1d20f1a674cb2" candidate_uuid: "bd26a06992144373b508bdb120fe2a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" is_pre_election: true
I20260621 05:19:51.749636 19966 raft_consensus.cc:2454] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd26a06992144373b508bdb120fe2a34 in term 0.
I20260621 05:19:51.749657 19832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e30713709e149e782e1d20f1a674cb2" candidate_uuid: "bd26a06992144373b508bdb120fe2a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b42dd2daafa449868f37518e7fb54011" is_pre_election: true
I20260621 05:19:51.749821 19832 raft_consensus.cc:2454] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd26a06992144373b508bdb120fe2a34 in term 0.
I20260621 05:19:51.749891 19631 leader_election.cc:302] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4307b5e7214d16a78cc89b90b3a2f0, bd26a06992144373b508bdb120fe2a34; no voters:
I20260621 05:19:51.750021 20037 raft_consensus.cc:2781] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:19:51.750077 20037 raft_consensus.cc:482] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:19:51.750093 20037 raft_consensus.cc:3037] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:51.750744 20037 raft_consensus.cc:504] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
I20260621 05:19:51.750890 20037 leader_election.cc:288] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [CANDIDATE]: Term 1 election: Requested vote from peers 2f4307b5e7214d16a78cc89b90b3a2f0 (127.0.98.3:41975), b42dd2daafa449868f37518e7fb54011 (127.0.98.2:39617)
I20260621 05:19:51.751045 19832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e30713709e149e782e1d20f1a674cb2" candidate_uuid: "bd26a06992144373b508bdb120fe2a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b42dd2daafa449868f37518e7fb54011"
I20260621 05:19:51.751065 19966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e30713709e149e782e1d20f1a674cb2" candidate_uuid: "bd26a06992144373b508bdb120fe2a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0"
I20260621 05:19:51.751125 19966 raft_consensus.cc:3037] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:51.751125 19832 raft_consensus.cc:3037] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:51.752004 19966 raft_consensus.cc:2454] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd26a06992144373b508bdb120fe2a34 in term 1.
I20260621 05:19:51.752004 19832 raft_consensus.cc:2454] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd26a06992144373b508bdb120fe2a34 in term 1.
I20260621 05:19:51.752177 19634 leader_election.cc:302] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b42dd2daafa449868f37518e7fb54011, bd26a06992144373b508bdb120fe2a34; no voters:
I20260621 05:19:51.752279 20037 raft_consensus.cc:2781] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:51.752436 20037 raft_consensus.cc:686] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [term 1 LEADER]: Becoming Leader. State: Replica: bd26a06992144373b508bdb120fe2a34, State: Running, Role: LEADER
I20260621 05:19:51.752573 20037 consensus_queue.cc:237] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } }
I20260621 05:19:51.753425 19553 catalog_manager.cc:5495] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd26a06992144373b508bdb120fe2a34 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bd26a06992144373b508bdb120fe2a34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd26a06992144373b508bdb120fe2a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } health_report { overall_health: UNKNOWN } } }
I20260621 05:19:51.762852 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.760495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.0.98.1:41267)
0621 05:19:51.760556 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.760558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.760567 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:51.760639 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.760640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.760658 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.760726 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:19:51.760883 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.761322 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.762107 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.762305 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.762377 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.762537 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:19:51.762631 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:19:51.762722 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.762741 (+ 19us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.762759 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:19:51.764791 19966 raft_consensus.cc:1261] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 1 FOLLOWER]: Refusing update from remote peer bd26a06992144373b508bdb120fe2a34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:51.764818 19832 raft_consensus.cc:1261] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 1 FOLLOWER]: Refusing update from remote peer bd26a06992144373b508bdb120fe2a34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:51.765115 20037 consensus_queue.cc:1035] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b42dd2daafa449868f37518e7fb54011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:51.765228 20039 consensus_queue.cc:1035] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4307b5e7214d16a78cc89b90b3a2f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:51.767453 20042 mvcc.cc:205] Tried to move back new op lower bound from 7299150609466212352 to 7299150609419423744. Current Snapshot: MvccSnapshot[applied={T|T < 7299150609466212352}]
I20260621 05:19:51.767758 20043 mvcc.cc:205] Tried to move back new op lower bound from 7299150609466212352 to 7299150609419423744. Current Snapshot: MvccSnapshot[applied={T|T < 7299150609466212352}]
I20260621 05:19:51.769490 19552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:51.769599 19552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:51.770341 19552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ed5c07033bf419b86a2f22026a47e65: Sending DeleteTablet for 3 replicas of tablet 2e30713709e149e782e1d20f1a674cb2
I20260621 05:19:51.770684 19946 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e30713709e149e782e1d20f1a674cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43954
I20260621 05:19:51.770671 19812 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e30713709e149e782e1d20f1a674cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55376
I20260621 05:19:51.770743 19678 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e30713709e149e782e1d20f1a674cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47652
I20260621 05:19:51.770961 20051 tablet_replica.cc:330] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: stopping tablet replica
I20260621 05:19:51.771057 20051 raft_consensus.cc:2229] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:51.771087 20053 tablet_replica.cc:330] T 2e30713709e149e782e1d20f1a674cb2 P bd26a06992144373b508bdb120fe2a34: stopping tablet replica
I20260621 05:19:51.771155 20051 raft_consensus.cc:2258] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:51.771230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 19615
I20260621 05:19:51.771942 20052 tablet_replica.cc:330] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: stopping tablet replica
I20260621 05:19:51.772081 20052 raft_consensus.cc:2229] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:51.772183 20052 raft_consensus.cc:2258] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:51.772435 20051 ts_tablet_manager.cc:1905] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:51.772992 20052 ts_tablet_manager.cc:1905] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:51.774034 20051 ts_tablet_manager.cc:1918] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:51.774108 20051 log.cc:1196] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/2e30713709e149e782e1d20f1a674cb2
I20260621 05:19:51.774406 20051 ts_tablet_manager.cc:1939] T 2e30713709e149e782e1d20f1a674cb2 P 2f4307b5e7214d16a78cc89b90b3a2f0: Deleting consensus metadata
I20260621 05:19:51.774653 20052 ts_tablet_manager.cc:1918] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:51.774724 20052 log.cc:1196] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/2e30713709e149e782e1d20f1a674cb2
I20260621 05:19:51.774820 19537 catalog_manager.cc:4841] TS 2f4307b5e7214d16a78cc89b90b3a2f0 (127.0.98.3:41975): tablet 2e30713709e149e782e1d20f1a674cb2 (table test-table [id=80ce81936f5144a48399b69c9ecfcd0c]) successfully deleted
I20260621 05:19:51.775027 20052 ts_tablet_manager.cc:1939] T 2e30713709e149e782e1d20f1a674cb2 P b42dd2daafa449868f37518e7fb54011: Deleting consensus metadata
I20260621 05:19:51.775422 19540 catalog_manager.cc:4841] TS b42dd2daafa449868f37518e7fb54011 (127.0.98.2:39617): tablet 2e30713709e149e782e1d20f1a674cb2 (table test-table [id=80ce81936f5144a48399b69c9ecfcd0c]) successfully deleted
W20260621 05:19:51.777228 19537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41267 (error 108)
I20260621 05:19:51.777707 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.777597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41267 (local address 127.0.0.1:47660)
0621 05:19:51.777648 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:51.777682 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 05:19:51.777840 19537 catalog_manager.cc:4584] TS bd26a06992144373b508bdb120fe2a34 (127.0.98.1:41267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e30713709e149e782e1d20f1a674cb2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41267: connect: Connection refused (error 111)
I20260621 05:19:51.777952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 19749
I20260621 05:19:51.783142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 19883
I20260621 05:19:51.788115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 19521
2026-06-21T05:19:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1978 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T05:19:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:51Z Disabled control of system clock
I20260621 05:19:51.814883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43529
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:19:51.892982 20058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:51.893201 20058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:51.893221 20058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:51.894272 20058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:51.894305 20058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:51.894318 20058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:51.894331 20058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:51.894342 20058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:51.895612 20058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:51.895812 20058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:51.896047 20058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:51.900650 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:51.900698 20058 server_base.cc:1034] running on GCE node
W20260621 05:19:51.900622 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:51.900640 20063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:51.901213 20058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:51.901454 20058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:51.902594 20058 hybrid_clock.cc:648] HybridClock initialized: now 1782019191902572 us; error 37 us; skew 500 ppm
I20260621 05:19:51.903820 20058 webserver.cc:460] Webserver started at http://127.0.98.62:40323/ using document root <none> and password file <none>
I20260621 05:19:51.904073 20058 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:51.904117 20058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:51.904266 20058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:51.905254 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "adb142e063f5419399a9a56b30eeb9f8"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "150bcc6be8b5b582165fa74db23ecc60"
server_key_iv: "f7b46c44248b97d3998aa49ab7c83872"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.905592 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "adb142e063f5419399a9a56b30eeb9f8"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "150bcc6be8b5b582165fa74db23ecc60"
server_key_iv: "f7b46c44248b97d3998aa49ab7c83872"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.906764 20058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:19:51.907483 20072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:51.907701 20058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:19:51.907799 20058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "adb142e063f5419399a9a56b30eeb9f8"
format_stamp: "Formatted at 2026-06-21 05:19:51 on dist-test-slave-czm5"
server_key: "150bcc6be8b5b582165fa74db23ecc60"
server_key_iv: "f7b46c44248b97d3998aa49ab7c83872"
server_key_version: "encryptionkey@0"
I20260621 05:19:51.907893 20058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:51.923087 20058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:51.923439 20058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:51.923588 20058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:51.927387 20058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43529
I20260621 05:19:51.927423 20124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43529 every 8 connection(s)
I20260621 05:19:51.927805 20058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:51.928690 20125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:51.928938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 20058
I20260621 05:19:51.929061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:51.929271 392 external_mini_cluster.cc:1428] Setting key 3f21e641c29f9fa83c758d679814e64a
I20260621 05:19:51.932307 20125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8: Bootstrap starting.
I20260621 05:19:51.932964 20125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:51.933269 20125 log.cc:824] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:51.933986 20125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8: No bootstrap required, opened a new log
I20260621 05:19:51.934993 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:51.929812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.0.98.62:43529)
0621 05:19:51.930115 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:51.930124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:51.930160 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:19:51.930597 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:51.930603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:51.930659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:51.930776 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:51.931036 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.932030 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.933142 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:51.933563 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:51.933649 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:51.934329 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:51.934333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:51.934333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:51.934523 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:51.934564 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:51.934577 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:51.934643 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:51.934701 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 05:19:51.934780 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260621 05:19:51.936205 20125 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb142e063f5419399a9a56b30eeb9f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } }
I20260621 05:19:51.936336 20125 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:51.936357 20125 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb142e063f5419399a9a56b30eeb9f8, State: Initialized, Role: FOLLOWER
I20260621 05:19:51.936465 20125 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb142e063f5419399a9a56b30eeb9f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } }
I20260621 05:19:51.936532 20125 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:51.936554 20125 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:51.936580 20125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:51.937070 20125 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb142e063f5419399a9a56b30eeb9f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } }
I20260621 05:19:51.937150 20125 leader_election.cc:302] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb142e063f5419399a9a56b30eeb9f8; no voters:
I20260621 05:19:51.937278 20125 leader_election.cc:288] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:51.937354 20130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:51.937536 20130 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [term 1 LEADER]: Becoming Leader. State: Replica: adb142e063f5419399a9a56b30eeb9f8, State: Running, Role: LEADER
I20260621 05:19:51.937600 20125 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:51.937625 20130 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb142e063f5419399a9a56b30eeb9f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } }
I20260621 05:19:51.937973 20131 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb142e063f5419399a9a56b30eeb9f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb142e063f5419399a9a56b30eeb9f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } } }
I20260621 05:19:51.937985 20132 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb142e063f5419399a9a56b30eeb9f8. Latest consensus state: current_term: 1 leader_uuid: "adb142e063f5419399a9a56b30eeb9f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb142e063f5419399a9a56b30eeb9f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } } }
I20260621 05:19:51.938162 20132 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:51.938122 20131 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:51.938552 20138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:51.939020 20138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:51.940562 20138 catalog_manager.cc:1347] Generated new cluster ID: fdf119a83d2543f596ca2c1ff4d25241
I20260621 05:19:51.940615 20138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:51.951987 20138 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:51.952471 20138 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:51.958132 20138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8: Generated new TSK 0
I20260621 05:19:51.958295 20138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:19:51.964321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--builtin_ntp_servers=127.0.98.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:19:52.045385 20149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:52.045585 20149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:52.045606 20149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:52.046684 20149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:52.046721 20149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:52.046744 20149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260621 05:19:52.048187 20149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:52.048445 20149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:52.048789 20149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:52.053512 20157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:52.053547 20154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:52.053522 20155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:52.054131 20149 server_base.cc:1034] running on GCE node
I20260621 05:19:52.054363 20149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:52.054630 20149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:52.055800 20149 hybrid_clock.cc:648] HybridClock initialized: now 1782019192055785 us; error 32 us; skew 500 ppm
I20260621 05:19:52.060792 20149 webserver.cc:460] Webserver started at http://127.0.98.1:33001/ using document root <none> and password file <none>
I20260621 05:19:52.061100 20149 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:52.061161 20149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:52.061305 20149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:52.062788 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/data/instance:
uuid: "213303036585469fa01c456009ee51d4"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "4d95ae31042de31a2ff4700e86cf3782"
server_key_iv: "7031ae2926e6e1642e3c84a43dece665"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.063192 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/wal/instance:
uuid: "213303036585469fa01c456009ee51d4"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "4d95ae31042de31a2ff4700e86cf3782"
server_key_iv: "7031ae2926e6e1642e3c84a43dece665"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.065517 20149 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260621 05:19:52.067739 20163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.067938 20149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260621 05:19:52.068053 20149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/wal
uuid: "213303036585469fa01c456009ee51d4"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "4d95ae31042de31a2ff4700e86cf3782"
server_key_iv: "7031ae2926e6e1642e3c84a43dece665"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.068142 20149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:52.086902 20149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:52.087275 20149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:52.087424 20149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:52.087687 20149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:52.088023 20149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:52.088068 20149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.088107 20149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:52.088132 20149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.096153 20149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42249
I20260621 05:19:52.096302 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42249 every 8 connection(s)
I20260621 05:19:52.096628 20149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/data/info.pb
I20260621 05:19:52.099372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 20149
I20260621 05:19:52.099493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-0/wal/instance
I20260621 05:19:52.099797 392 external_mini_cluster.cc:1428] Setting key 67bf841b2e07c93005de5a24ace51da8
I20260621 05:19:52.100838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--builtin_ntp_servers=127.0.98.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:19:52.102694 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.097390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35329 (local address 127.0.98.62:43529)
0621 05:19:52.097459 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.097461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.097743 (+ 282us) server_negotiation.cc:407] Connection header received
0621 05:19:52.098291 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.098292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.098307 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.098361 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:52.099082 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.099879 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.101074 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.101402 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.101478 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.102395 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:52.102397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:52.102397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:52.102401 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:52.102414 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:52.102418 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:52.102461 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.102557 (+ 96us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.102609 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:19:52.102902 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.097399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43529 (local address 127.0.98.1:35329)
0621 05:19:52.097615 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.097647 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:52.098191 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:52.098412 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:52.098415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:52.098422 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:52.098918 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.098922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.099992 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.099994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.100960 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.100963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.101489 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.101490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.101611 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.102229 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:52.102230 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:52.102272 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:52.102305 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:52.102483 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:52.102487 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:52.102490 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:52.102559 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:19:52.102663 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260621 05:19:52.103389 20277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43529
I20260621 05:19:52.103480 20277 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:52.103691 20277 heartbeater.cc:511] Master 127.0.98.62:43529 requested a full tablet report, sending...
I20260621 05:19:52.104254 20089 ts_manager.cc:194] Registered new tserver with Master: 213303036585469fa01c456009ee51d4 (127.0.98.1:42249)
I20260621 05:19:52.104916 20089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35329
W20260621 05:19:52.189070 20280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:52.189280 20280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:52.189301 20280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:52.190480 20280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:52.190522 20280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:52.190547 20280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260621 05:19:52.192022 20280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:52.192273 20280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:52.192571 20280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:52.197628 20286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:52.197643 20288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:52.197712 20280 server_base.cc:1034] running on GCE node
W20260621 05:19:52.197654 20285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:52.198072 20280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:52.198375 20280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:52.199523 20280 hybrid_clock.cc:648] HybridClock initialized: now 1782019192199513 us; error 32 us; skew 500 ppm
I20260621 05:19:52.200912 20280 webserver.cc:460] Webserver started at http://127.0.98.2:41097/ using document root <none> and password file <none>
I20260621 05:19:52.201119 20280 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:52.201159 20280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:52.201243 20280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:52.202260 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/data/instance:
uuid: "05be6d3e08eb4f2e9411d5bf479c3532"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "8f5fe5ba807b58a5e40f7f05ae947917"
server_key_iv: "c7232c4846ea5b43e13ada77301aeda6"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.202564 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05be6d3e08eb4f2e9411d5bf479c3532"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "8f5fe5ba807b58a5e40f7f05ae947917"
server_key_iv: "c7232c4846ea5b43e13ada77301aeda6"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.203894 20280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:19:52.204566 20294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.204756 20280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:52.204850 20280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/wal
uuid: "05be6d3e08eb4f2e9411d5bf479c3532"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "8f5fe5ba807b58a5e40f7f05ae947917"
server_key_iv: "c7232c4846ea5b43e13ada77301aeda6"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.204926 20280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:52.222440 20280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:52.222783 20280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:52.222951 20280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:52.223182 20280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:52.223527 20280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:52.223582 20280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.223616 20280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:52.223680 20280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.229326 20280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46709
I20260621 05:19:52.229410 20407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46709 every 8 connection(s)
I20260621 05:19:52.229763 20280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/data/info.pb
I20260621 05:19:52.235074 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.230545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59151 (local address 127.0.98.62:43529)
0621 05:19:52.230607 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.230608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.230864 (+ 256us) server_negotiation.cc:407] Connection header received
0621 05:19:52.231435 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.231437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.231450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.231497 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:52.232251 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.232708 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.233716 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.233913 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.234003 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.234765 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:52.234766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:52.234766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:52.234771 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:52.234785 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:52.234789 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:52.234843 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.234960 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.234992 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:19:52.235301 20410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.230521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43529 (local address 127.0.98.2:59151)
0621 05:19:52.230730 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.230770 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:52.231303 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:52.231583 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:52.231591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:52.231600 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:52.232147 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.232152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.232826 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.232827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.233612 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.233614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.234012 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.234013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.234136 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.234633 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:52.234633 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:52.234658 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:52.234678 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:52.234858 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:52.234868 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:52.234871 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:52.234963 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:19:52.235062 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260621 05:19:52.235908 20408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43529
I20260621 05:19:52.235987 20408 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:52.236310 20408 heartbeater.cc:511] Master 127.0.98.62:43529 requested a full tablet report, sending...
I20260621 05:19:52.236591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 20280
I20260621 05:19:52.236714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/wal/instance
I20260621 05:19:52.236783 20089 ts_manager.cc:194] Registered new tserver with Master: 05be6d3e08eb4f2e9411d5bf479c3532 (127.0.98.2:46709)
I20260621 05:19:52.236941 392 external_mini_cluster.cc:1428] Setting key a575cf90aa51728fce25552f84be533d
I20260621 05:19:52.237483 20089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59151
I20260621 05:19:52.238035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--builtin_ntp_servers=127.0.98.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:19:52.317569 20411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:52.317780 20411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:52.317802 20411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:52.318979 20411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:52.319021 20411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:52.319046 20411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260621 05:19:52.320533 20411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:52.320799 20411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:52.321105 20411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:19:52.326072 20411 server_base.cc:1034] running on GCE node
W20260621 05:19:52.325981 20416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:52.326000 20417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:52.325981 20419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:52.326495 20411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:52.326776 20411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:52.327914 20411 hybrid_clock.cc:648] HybridClock initialized: now 1782019192327902 us; error 29 us; skew 500 ppm
I20260621 05:19:52.329073 20411 webserver.cc:460] Webserver started at http://127.0.98.3:43683/ using document root <none> and password file <none>
I20260621 05:19:52.329331 20411 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:52.329376 20411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:52.329536 20411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:52.330608 20411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/data/instance:
uuid: "6e65f52e51884c86b6847bf4c83cb837"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "24b983aa0af440a3a63cd29a3ce814e7"
server_key_iv: "d02b9a3ac656fa7659be27b5f29b2a00"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.330960 20411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6e65f52e51884c86b6847bf4c83cb837"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "24b983aa0af440a3a63cd29a3ce814e7"
server_key_iv: "d02b9a3ac656fa7659be27b5f29b2a00"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.332293 20411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:19:52.332966 20425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.333132 20411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:52.333212 20411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/wal
uuid: "6e65f52e51884c86b6847bf4c83cb837"
format_stamp: "Formatted at 2026-06-21 05:19:52 on dist-test-slave-czm5"
server_key: "24b983aa0af440a3a63cd29a3ce814e7"
server_key_iv: "d02b9a3ac656fa7659be27b5f29b2a00"
server_key_version: "encryptionkey@0"
I20260621 05:19:52.333280 20411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:52.346635 20411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:52.346967 20411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:52.347110 20411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:52.347332 20411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:19:52.347700 20411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:19:52.347751 20411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.347798 20411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:19:52.347839 20411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:52.353586 20411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34265
I20260621 05:19:52.353622 20538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34265 every 8 connection(s)
I20260621 05:19:52.353933 20411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/data/info.pb
I20260621 05:19:52.358709 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.354639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41593 (local address 127.0.98.62:43529)
0621 05:19:52.354685 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.354687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.354923 (+ 236us) server_negotiation.cc:407] Connection header received
0621 05:19:52.355352 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.355353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.355362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.355412 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:52.356089 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.356554 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.357504 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.357681 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.357727 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.358503 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:52.358504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:52.358504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:52.358507 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:52.358517 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:52.358520 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:52.358556 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.358636 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.358652 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:19:52.358913 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.354637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43529 (local address 127.0.98.3:41593)
0621 05:19:52.354804 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.354843 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:52.355279 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:52.355460 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:52.355462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:52.355469 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:19:52.356017 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.356022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.356628 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.356628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.357433 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.357435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.357747 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.357748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.357855 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.358388 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:19:52.358388 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:19:52.358415 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:19:52.358436 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:19:52.358571 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:19:52.358575 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:19:52.358577 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:19:52.358628 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:19:52.358700 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260621 05:19:52.359373 20539 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43529
I20260621 05:19:52.359455 20539 heartbeater.cc:464] Registering TS with master...
I20260621 05:19:52.359582 20539 heartbeater.cc:511] Master 127.0.98.62:43529 requested a full tablet report, sending...
I20260621 05:19:52.359964 20088 ts_manager.cc:194] Registered new tserver with Master: 6e65f52e51884c86b6847bf4c83cb837 (127.0.98.3:34265)
I20260621 05:19:52.360292 20088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41593
I20260621 05:19:52.363142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 20411
I20260621 05:19:52.363235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/wal/instance
I20260621 05:19:52.363363 392 external_mini_cluster.cc:1428] Setting key 0e93a98020de6a898c16f8b016c23ecd
I20260621 05:19:52.364184 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:19:52.368486 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.365441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.0.98.62:43529)
0621 05:19:52.365477 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.365479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.365651 (+ 172us) server_negotiation.cc:407] Connection header received
0621 05:19:52.365745 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.365746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.365755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.365843 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:52.366022 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.366449 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.367222 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.367411 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.367496 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.368243 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:52.368245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:52.368245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:52.368250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:52.368263 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:52.368267 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:52.368318 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.368383 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.368404 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:19:52.369971 20088 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:19:52.371482 20088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:19:52.377962 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.373867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.0.98.1:42249)
0621 05:19:52.374151 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.374154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.374365 (+ 211us) server_negotiation.cc:407] Connection header received
0621 05:19:52.374500 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.374503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.374517 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.374579 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:52.374758 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.375799 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.377061 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.377545 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.377636 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.377722 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.377816 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.377865 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":94,"threads_started":1}
I20260621 05:19:52.377967 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.373770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.0.1:36250)
0621 05:19:52.374302 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.374310 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:52.374410 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:52.374642 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:52.374642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:52.374644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:52.374676 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.374678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.375875 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.375876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.376972 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.376974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.377638 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.377639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.377710 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.377723 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:19:52.377787 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 05:19:52.377896 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":32,"thread_start_us":76,"threads_started":1}
I20260621 05:19:52.378095 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.373686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.0.98.3:34265)
0621 05:19:52.374073 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.374078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.374194 (+ 116us) server_negotiation.cc:407] Connection header received
0621 05:19:52.374259 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.374261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.374275 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.374339 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:52.374520 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.375203 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.376820 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.377560 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.377634 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.377722 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.377978 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.378017 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":90,"threads_started":1}
I20260621 05:19:52.378110 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.373872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.0.98.2:46709)
0621 05:19:52.374039 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.374043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.374052 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:19:52.374160 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.374162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.374177 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.374260 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:52.374432 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.375551 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.376910 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.377572 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.377634 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.377722 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.378004 (+ 282us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.378039 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260621 05:19:52.378289 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.373574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34265 (local address 127.0.0.1:56112)
0621 05:19:52.374135 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.374145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:52.374192 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:52.374397 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:52.374398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:52.374400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:52.374445 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.374447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.375329 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.375329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.376711 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.376714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.377762 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.377762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.377890 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.377894 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:52.378191 (+ 297us) client_negotiation.cc:240] Negotiation successful
0621 05:19:52.378230 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":50,"thread_start_us":67,"threads_started":1}
I20260621 05:19:52.378680 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.373574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46709 (local address 127.0.0.1:36562)
0621 05:19:52.373716 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.373757 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:52.374030 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:52.374277 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:52.374281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:52.374283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:52.374352 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.374353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.375662 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.375663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.376840 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.376843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.377759 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.377761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.377850 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.377853 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:19:52.377914 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:19:52.377962 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260621 05:19:52.379014 20211 tablet_service.cc:1467] Processing CreateTablet for tablet dae15f0789984ed2a5ce7c609b299ab0 (DEFAULT_TABLE table=test-table [id=e4814102f3104fbd9e18a55fd45bae6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:52.379014 20473 tablet_service.cc:1467] Processing CreateTablet for tablet dae15f0789984ed2a5ce7c609b299ab0 (DEFAULT_TABLE table=test-table [id=e4814102f3104fbd9e18a55fd45bae6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:52.379369 20473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae15f0789984ed2a5ce7c609b299ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:52.379369 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae15f0789984ed2a5ce7c609b299ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:52.382774 20553 tablet_bootstrap.cc:492] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4: Bootstrap starting.
I20260621 05:19:52.382966 20554 tablet_bootstrap.cc:492] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: Bootstrap starting.
I20260621 05:19:52.383464 20342 tablet_service.cc:1467] Processing CreateTablet for tablet dae15f0789984ed2a5ce7c609b299ab0 (DEFAULT_TABLE table=test-table [id=e4814102f3104fbd9e18a55fd45bae6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:19:52.383548 20554 tablet_bootstrap.cc:654] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:52.383548 20553 tablet_bootstrap.cc:654] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:52.383805 20342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae15f0789984ed2a5ce7c609b299ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:52.383899 20554 log.cc:824] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:52.383962 20553 log.cc:824] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:52.384613 20554 tablet_bootstrap.cc:492] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: No bootstrap required, opened a new log
I20260621 05:19:52.384682 20554 ts_tablet_manager.cc:1397] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:19:52.386166 20553 tablet_bootstrap.cc:492] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4: No bootstrap required, opened a new log
I20260621 05:19:52.386237 20553 ts_tablet_manager.cc:1397] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 05:19:52.386611 20557 tablet_bootstrap.cc:492] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Bootstrap starting.
I20260621 05:19:52.386686 20554 raft_consensus.cc:348] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.386902 20554 raft_consensus.cc:374] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:52.386942 20554 raft_consensus.cc:729] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e65f52e51884c86b6847bf4c83cb837, State: Initialized, Role: FOLLOWER
I20260621 05:19:52.387075 20554 consensus_queue.cc:260] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.387313 20554 ts_tablet_manager.cc:1428] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:19:52.387337 20557 tablet_bootstrap.cc:654] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:52.387480 20539 heartbeater.cc:503] Master 127.0.98.62:43529 was elected leader, sending a full tablet report...
I20260621 05:19:52.387689 20557 log.cc:824] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:52.388532 20557 tablet_bootstrap.cc:492] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: No bootstrap required, opened a new log
I20260621 05:19:52.388608 20557 ts_tablet_manager.cc:1397] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:19:52.388722 20553 raft_consensus.cc:348] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.388857 20553 raft_consensus.cc:374] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:52.388880 20553 raft_consensus.cc:729] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213303036585469fa01c456009ee51d4, State: Initialized, Role: FOLLOWER
I20260621 05:19:52.389001 20553 consensus_queue.cc:260] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.389259 20553 ts_tablet_manager.cc:1428] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:19:52.389293 20277 heartbeater.cc:503] Master 127.0.98.62:43529 was elected leader, sending a full tablet report...
I20260621 05:19:52.391023 20557 raft_consensus.cc:348] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.391166 20557 raft_consensus.cc:374] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:52.391228 20557 raft_consensus.cc:729] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05be6d3e08eb4f2e9411d5bf479c3532, State: Initialized, Role: FOLLOWER
I20260621 05:19:52.391338 20557 consensus_queue.cc:260] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.391592 20557 ts_tablet_manager.cc:1428] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:19:52.391664 20408 heartbeater.cc:503] Master 127.0.98.62:43529 was elected leader, sending a full tablet report...
W20260621 05:19:52.480334 20409 tablet.cc:2369] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:19:52.506983 20558 raft_consensus.cc:482] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:19:52.507110 20558 raft_consensus.cc:504] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.507545 20558 leader_election.cc:288] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05be6d3e08eb4f2e9411d5bf479c3532 (127.0.98.2:46709), 213303036585469fa01c456009ee51d4 (127.0.98.1:42249)
I20260621 05:19:52.510634 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.507865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33171 (local address 127.0.98.2:46709)
0621 05:19:52.507903 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.507905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.507916 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:19:52.507974 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.507975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.507985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.508056 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:52.508257 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.508793 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.509859 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.510275 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.510370 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.510380 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.510526 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.510561 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:19:52.510645 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.507800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46709 (local address 127.0.98.3:33171)
0621 05:19:52.507859 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.507870 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:52.507914 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:52.508073 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:52.508074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:52.508075 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:52.508188 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.508190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.508905 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.508906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.509734 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.509736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.510372 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.510373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.510444 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.510450 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:19:52.510519 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:19:52.510562 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":3}
I20260621 05:19:52.510664 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.507933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.98.3:58249)
0621 05:19:52.508073 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.508083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:19:52.508135 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:19:52.508257 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:19:52.508258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:19:52.508259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:19:52.508318 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.508319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.508958 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.508959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.509756 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:19:52.509757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.510375 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.510376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:19:52.510458 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.510462 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:19:52.510530 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:19:52.510571 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260621 05:19:52.510658 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.507984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58249 (local address 127.0.98.1:42249)
0621 05:19:52.508031 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.508033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.508149 (+ 116us) server_negotiation.cc:407] Connection header received
0621 05:19:52.508180 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.508181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.508192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.508244 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:19:52.508368 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.508859 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.509866 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.510292 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.510375 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.510384 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.510560 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.510598 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:19:52.511067 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae15f0789984ed2a5ce7c609b299ab0" candidate_uuid: "6e65f52e51884c86b6847bf4c83cb837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213303036585469fa01c456009ee51d4" is_pre_election: true
I20260621 05:19:52.511238 20231 raft_consensus.cc:2454] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e65f52e51884c86b6847bf4c83cb837 in term 0.
I20260621 05:19:52.511479 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae15f0789984ed2a5ce7c609b299ab0" candidate_uuid: "6e65f52e51884c86b6847bf4c83cb837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" is_pre_election: true
I20260621 05:19:52.511518 20427 leader_election.cc:302] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213303036585469fa01c456009ee51d4, 6e65f52e51884c86b6847bf4c83cb837; no voters:
I20260621 05:19:52.511655 20558 raft_consensus.cc:2781] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:19:52.511688 20362 raft_consensus.cc:2454] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e65f52e51884c86b6847bf4c83cb837 in term 0.
I20260621 05:19:52.511711 20558 raft_consensus.cc:482] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:19:52.511765 20558 raft_consensus.cc:3037] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:52.512466 20558 raft_consensus.cc:504] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.512600 20558 leader_election.cc:288] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [CANDIDATE]: Term 1 election: Requested vote from peers 05be6d3e08eb4f2e9411d5bf479c3532 (127.0.98.2:46709), 213303036585469fa01c456009ee51d4 (127.0.98.1:42249)
I20260621 05:19:52.512768 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae15f0789984ed2a5ce7c609b299ab0" candidate_uuid: "6e65f52e51884c86b6847bf4c83cb837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05be6d3e08eb4f2e9411d5bf479c3532"
I20260621 05:19:52.512814 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae15f0789984ed2a5ce7c609b299ab0" candidate_uuid: "6e65f52e51884c86b6847bf4c83cb837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213303036585469fa01c456009ee51d4"
I20260621 05:19:52.512863 20362 raft_consensus.cc:3037] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:52.512904 20231 raft_consensus.cc:3037] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:52.513566 20362 raft_consensus.cc:2454] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e65f52e51884c86b6847bf4c83cb837 in term 1.
I20260621 05:19:52.513614 20231 raft_consensus.cc:2454] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e65f52e51884c86b6847bf4c83cb837 in term 1.
I20260621 05:19:52.513788 20427 leader_election.cc:302] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213303036585469fa01c456009ee51d4, 6e65f52e51884c86b6847bf4c83cb837; no voters:
I20260621 05:19:52.513901 20558 raft_consensus.cc:2781] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:52.514056 20558 raft_consensus.cc:686] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 1 LEADER]: Becoming Leader. State: Replica: 6e65f52e51884c86b6847bf4c83cb837, State: Running, Role: LEADER
I20260621 05:19:52.514153 20558 consensus_queue.cc:237] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } }
I20260621 05:19:52.514962 20088 catalog_manager.cc:5495] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e65f52e51884c86b6847bf4c83cb837 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6e65f52e51884c86b6847bf4c83cb837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e65f52e51884c86b6847bf4c83cb837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34265 } health_report { overall_health: HEALTHY } } }
I20260621 05:19:52.525907 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.523523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.0.98.3:34265)
0621 05:19:52.523587 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:52.523589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:52.523601 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:19:52.523656 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:52.523657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:52.523675 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:52.523758 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:19:52.523890 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.524359 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.525086 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:52.525278 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:52.525351 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:52.525520 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:19:52.525638 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:19:52.525731 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:52.525749 (+ 18us) server_negotiation.cc:299] Negotiation successful
0621 05:19:52.525790 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:19:52.527704 20362 raft_consensus.cc:1261] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 1 FOLLOWER]: Refusing update from remote peer 6e65f52e51884c86b6847bf4c83cb837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:52.527757 20231 raft_consensus.cc:1261] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 1 FOLLOWER]: Refusing update from remote peer 6e65f52e51884c86b6847bf4c83cb837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:19:52.528059 20558 consensus_queue.cc:1035] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05be6d3e08eb4f2e9411d5bf479c3532" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:52.528182 20563 consensus_queue.cc:1035] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213303036585469fa01c456009ee51d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:19:52.530520 20566 mvcc.cc:205] Tried to move back new op lower bound from 7299150612590825472 to 7299150612538654720. Current Snapshot: MvccSnapshot[applied={T|T < 7299150612590825472}]
I20260621 05:19:52.530552 20568 mvcc.cc:205] Tried to move back new op lower bound from 7299150612590825472 to 7299150612538654720. Current Snapshot: MvccSnapshot[applied={T|T < 7299150612590825472}]
I20260621 05:19:52.531596 20567 mvcc.cc:205] Tried to move back new op lower bound from 7299150612590825472 to 7299150612538654720. Current Snapshot: MvccSnapshot[applied={T|T < 7299150612590825472 or (T in {7299150612590825472})}]
I20260621 05:19:52.533070 20088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:52.533144 20088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:19:52.533783 20088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adb142e063f5419399a9a56b30eeb9f8: Sending DeleteTablet for 3 replicas of tablet dae15f0789984ed2a5ce7c609b299ab0
I20260621 05:19:52.534135 20473 tablet_service.cc:1514] Processing DeleteTablet for tablet dae15f0789984ed2a5ce7c609b299ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:52 UTC) from {username='slave'} at 127.0.0.1:56112
I20260621 05:19:52.534197 20342 tablet_service.cc:1514] Processing DeleteTablet for tablet dae15f0789984ed2a5ce7c609b299ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:52 UTC) from {username='slave'} at 127.0.0.1:36562
I20260621 05:19:52.534255 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet dae15f0789984ed2a5ce7c609b299ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:19:52 UTC) from {username='slave'} at 127.0.0.1:36250
I20260621 05:19:52.534438 20576 tablet_replica.cc:330] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: stopping tablet replica
I20260621 05:19:52.534479 20578 tablet_replica.cc:330] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4: stopping tablet replica
I20260621 05:19:52.534598 20578 raft_consensus.cc:2229] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:52.534677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 20149
I20260621 05:19:52.534628 20576 raft_consensus.cc:2229] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:19:52.534701 20578 raft_consensus.cc:2258] T dae15f0789984ed2a5ce7c609b299ab0 P 213303036585469fa01c456009ee51d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:52.535413 20576 raft_consensus.cc:2258] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:52.535650 20577 tablet_replica.cc:330] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: stopping tablet replica
I20260621 05:19:52.535807 20577 raft_consensus.cc:2229] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:19:52.535961 20577 raft_consensus.cc:2258] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:19:52.536454 20577 ts_tablet_manager.cc:1905] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:52.537185 20576 ts_tablet_manager.cc:1905] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:19:52.538388 20577 ts_tablet_manager.cc:1918] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:52.538517 20577 log.cc:1196] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-1/wal/wals/dae15f0789984ed2a5ce7c609b299ab0
I20260621 05:19:52.538841 20577 ts_tablet_manager.cc:1939] T dae15f0789984ed2a5ce7c609b299ab0 P 05be6d3e08eb4f2e9411d5bf479c3532: Deleting consensus metadata
I20260621 05:19:52.539018 20576 ts_tablet_manager.cc:1918] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:19:52.539098 20576 log.cc:1196] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: Deleting WAL directory at /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018933478224-392-0/minicluster-data/ts-2/wal/wals/dae15f0789984ed2a5ce7c609b299ab0
I20260621 05:19:52.539350 20576 ts_tablet_manager.cc:1939] T dae15f0789984ed2a5ce7c609b299ab0 P 6e65f52e51884c86b6847bf4c83cb837: Deleting consensus metadata
I20260621 05:19:52.539465 20074 catalog_manager.cc:4841] TS 05be6d3e08eb4f2e9411d5bf479c3532 (127.0.98.2:46709): tablet dae15f0789984ed2a5ce7c609b299ab0 (table test-table [id=e4814102f3104fbd9e18a55fd45bae6e]) successfully deleted
I20260621 05:19:52.539721 20075 catalog_manager.cc:4841] TS 6e65f52e51884c86b6847bf4c83cb837 (127.0.98.3:34265): tablet dae15f0789984ed2a5ce7c609b299ab0 (table test-table [id=e4814102f3104fbd9e18a55fd45bae6e]) successfully deleted
W20260621 05:19:52.542333 20074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42249 (error 108)
I20260621 05:19:52.542667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 20280
I20260621 05:19:52.543983 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:52.543817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.0.1:37344)
0621 05:19:52.543893 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0621 05:19:52.543947 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 05:19:52.544133 20074 catalog_manager.cc:4584] TS 213303036585469fa01c456009ee51d4 (127.0.98.1:42249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dae15f0789984ed2a5ce7c609b299ab0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42249: connect: Connection refused (error 111)
I20260621 05:19:52.547417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 20411
I20260621 05:19:52.552196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 20058
2026-06-21T05:19:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (763 ms)
[----------] 6 tests from EncryptionPolicyTest (8026 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:52 dist-test-slave-czm5 krb5kdc[20582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:52 dist-test-slave-czm5 krb5kdc[20582](info): set up 2 sockets
Jun 21 05:19:52 dist-test-slave-czm5 krb5kdc[20582](info): commencing operation
krb5kdc: starting...
W20260621 05:19:53.575445 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:19:53 dist-test-slave-czm5 krb5kdc[20582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:19:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:19:53.687959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:53.766047 20595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:53.766292 20595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:53.766333 20595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:53.767454 20595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:53.767527 20595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:53.767554 20595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:53.767582 20595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:53.767609 20595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:53.769049 20595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:53.769335 20595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:53.769549 20595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:53.774287 20603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:53.774326 20595 server_base.cc:1034] running on GCE node
W20260621 05:19:53.774392 20601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:53.774562 20600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:53.774809 20595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:53.775141 20595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:53.776291 20595 hybrid_clock.cc:648] HybridClock initialized: now 1782019193776268 us; error 40 us; skew 500 ppm
Jun 21 05:19:53 dist-test-slave-czm5 krb5kdc[20582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:53.778687 20595 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:19:53.779054 20595 webserver.cc:460] Webserver started at http://127.0.0.1:37999/ using document root <none> and password file <none>
I20260621 05:19:53.779234 20595 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:53.779273 20595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:53.779357 20595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:53.780378 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "daccf0bbe698454e86981aee0336c98f"
format_stamp: "Formatted at 2026-06-21 05:19:53 on dist-test-slave-czm5"
server_key: "36de4d77f5536048b7bfa34770e68446"
server_key_iv: "7642643e635759eb12417d268ec2a16f"
server_key_version: "encryptionkey@0"
I20260621 05:19:53.780665 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "daccf0bbe698454e86981aee0336c98f"
format_stamp: "Formatted at 2026-06-21 05:19:53 on dist-test-slave-czm5"
server_key: "36de4d77f5536048b7bfa34770e68446"
server_key_iv: "7642643e635759eb12417d268ec2a16f"
server_key_version: "encryptionkey@0"
I20260621 05:19:53.781924 20595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:19:53.782531 20610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:53.782701 20595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:53.782783 20595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "daccf0bbe698454e86981aee0336c98f"
format_stamp: "Formatted at 2026-06-21 05:19:53 on dist-test-slave-czm5"
server_key: "36de4d77f5536048b7bfa34770e68446"
server_key_iv: "7642643e635759eb12417d268ec2a16f"
server_key_version: "encryptionkey@0"
I20260621 05:19:53.782835 20595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:53.793264 20595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:53.794899 20595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:53.795030 20595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:53.798740 20595 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42117
I20260621 05:19:53.798853 20662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42117 every 8 connection(s)
I20260621 05:19:53.799135 20595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:53.800062 20663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:53.802229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 20595
I20260621 05:19:53.802244 20663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f: Bootstrap starting.
I20260621 05:19:53.802349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:53.802558 392 external_mini_cluster.cc:1428] Setting key 1cf4675ddf794a629d95896d5accae6c
I20260621 05:19:53.802922 20663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:53.803491 20663 log.cc:824] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:53.804293 20663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f: No bootstrap required, opened a new log
I20260621 05:19:53.806576 20663 raft_consensus.cc:348] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daccf0bbe698454e86981aee0336c98f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42117 } }
I20260621 05:19:53.806707 20663 raft_consensus.cc:374] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:53.806730 20663 raft_consensus.cc:729] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daccf0bbe698454e86981aee0336c98f, State: Initialized, Role: FOLLOWER
I20260621 05:19:53.806856 20663 consensus_queue.cc:260] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daccf0bbe698454e86981aee0336c98f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42117 } }
I20260621 05:19:53.806927 20663 raft_consensus.cc:388] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:53.806947 20663 raft_consensus.cc:482] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:53.807013 20663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:53.807718 20663 raft_consensus.cc:504] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daccf0bbe698454e86981aee0336c98f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42117 } }
I20260621 05:19:53.807821 20663 leader_election.cc:302] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daccf0bbe698454e86981aee0336c98f; no voters:
I20260621 05:19:53.807979 20663 leader_election.cc:288] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:53.808143 20669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:53.808251 20663 sys_catalog.cc:564] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:53.808332 20669 raft_consensus.cc:686] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [term 1 LEADER]: Becoming Leader. State: Replica: daccf0bbe698454e86981aee0336c98f, State: Running, Role: LEADER
I20260621 05:19:53.808423 20669 consensus_queue.cc:237] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daccf0bbe698454e86981aee0336c98f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42117 } }
I20260621 05:19:53.808890 20668 sys_catalog.cc:455] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daccf0bbe698454e86981aee0336c98f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daccf0bbe698454e86981aee0336c98f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42117 } } }
I20260621 05:19:53.808990 20668 sys_catalog.cc:458] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:53.809121 20670 sys_catalog.cc:455] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [sys.catalog]: SysCatalogTable state changed. Reason: New leader daccf0bbe698454e86981aee0336c98f. Latest consensus state: current_term: 1 leader_uuid: "daccf0bbe698454e86981aee0336c98f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daccf0bbe698454e86981aee0336c98f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42117 } } }
I20260621 05:19:53.809190 20670 sys_catalog.cc:458] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:53.809284 20677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:53.809837 20677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:19:53 dist-test-slave-czm5 krb5kdc[20582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:19:53.811875 20677 catalog_manager.cc:1347] Generated new cluster ID: ffe5599067324b598ca237fae3322a17
I20260621 05:19:53.811909 20677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:53.814707 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:53.803091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45221 (local address 127.0.0.1:42117)
0621 05:19:53.803264 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:53.803273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:53.803312 (+ 39us) server_negotiation.cc:407] Connection header received
0621 05:19:53.803914 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:53.803921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:53.804019 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:53.804164 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:53.804874 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:53.806309 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:53.807240 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:53.807905 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:53.808026 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:53.810931 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:53.810936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:53.810937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:53.810949 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:53.812936 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:53.813414 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:53.813417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:53.813417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:53.813516 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:53.813787 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:53.813788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:53.813788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:53.814060 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:53.814189 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:53.814392 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 05:19:53.814485 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260621 05:19:53.815766 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:19:53.820588 20677 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:53.821225 20677 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:53.824600 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:53.817358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50717 (local address 127.0.0.1:42117)
0621 05:19:53.817398 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:53.817400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:53.817536 (+ 136us) server_negotiation.cc:407] Connection header received
0621 05:19:53.817563 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:53.817564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:53.817574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:53.817642 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:53.818218 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:53.818957 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:53.819948 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:53.820264 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:53.820332 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:53.821732 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:53.821733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:53.821734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:53.821741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:53.823381 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:53.823796 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:53.823797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:53.823797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:53.823868 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:53.824082 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:53.824083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:53.824083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:53.824204 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:53.824303 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:53.824413 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 05:19:53.824474 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":20}
I20260621 05:19:53.826180 20677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daccf0bbe698454e86981aee0336c98f: Generated new TSK 0
I20260621 05:19:53.826375 20677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:19:53.834786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 20595
W20260621 05:19:53.838889 20580 connection.cc:535] client connection to 127.0.0.1:42117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42117): Connection reset by peer (error 104)
2026-06-21T05:19:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:53 dist-test-slave-czm5 krb5kdc[20694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:53 dist-test-slave-czm5 krb5kdc[20694](info): set up 2 sockets
Jun 21 05:19:53 dist-test-slave-czm5 krb5kdc[20694](info): commencing operation
krb5kdc: starting...
W20260621 05:19:54.851459 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:19:54 dist-test-slave-czm5 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:19:54.962707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:55.041281 20707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:55.041491 20707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:55.041513 20707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:55.042649 20707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:55.042690 20707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:55.042704 20707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:55.042716 20707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:55.042770 20707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:55.044318 20707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:55.044566 20707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:55.044787 20707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:55.049716 20713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:55.049803 20712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:55.049758 20715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:55.049871 20707 server_base.cc:1034] running on GCE node
I20260621 05:19:55.050284 20707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:55.050573 20707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:55.051687 20707 hybrid_clock.cc:648] HybridClock initialized: now 1782019195051672 us; error 29 us; skew 500 ppm
Jun 21 05:19:55 dist-test-slave-czm5 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:55.058758 20707 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:19:55.059219 20707 webserver.cc:460] Webserver started at http://127.0.0.1:45239/ using document root <none> and password file <none>
I20260621 05:19:55.059449 20707 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:55.059518 20707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:55.059705 20707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:55.060819 20707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "1da69110180e4afaa46b01ac5ca19cb7"
format_stamp: "Formatted at 2026-06-21 05:19:55 on dist-test-slave-czm5"
server_key: "c7af33b2d416843a1b67d71c18a151ab"
server_key_iv: "48d1a52ae9ab954cb9a06d7540453aeb"
server_key_version: "encryptionkey@0"
I20260621 05:19:55.061196 20707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "1da69110180e4afaa46b01ac5ca19cb7"
format_stamp: "Formatted at 2026-06-21 05:19:55 on dist-test-slave-czm5"
server_key: "c7af33b2d416843a1b67d71c18a151ab"
server_key_iv: "48d1a52ae9ab954cb9a06d7540453aeb"
server_key_version: "encryptionkey@0"
I20260621 05:19:55.062523 20707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:19:55.063261 20722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:55.063434 20707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:19:55.063547 20707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "1da69110180e4afaa46b01ac5ca19cb7"
format_stamp: "Formatted at 2026-06-21 05:19:55 on dist-test-slave-czm5"
server_key: "c7af33b2d416843a1b67d71c18a151ab"
server_key_iv: "48d1a52ae9ab954cb9a06d7540453aeb"
server_key_version: "encryptionkey@0"
I20260621 05:19:55.063621 20707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:55.084694 20707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:55.086508 20707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:55.086671 20707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:55.090730 20707 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34751
I20260621 05:19:55.090768 20774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34751 every 8 connection(s)
I20260621 05:19:55.091142 20707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:55.092108 20775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:55.094377 20775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7: Bootstrap starting.
I20260621 05:19:55.094959 20775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:55.095243 20775 log.cc:824] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:55.095991 20775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7: No bootstrap required, opened a new log
I20260621 05:19:55.097465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 20707
I20260621 05:19:55.097553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:55.097554 20775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da69110180e4afaa46b01ac5ca19cb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } }
I20260621 05:19:55.097730 20775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:55.097770 392 external_mini_cluster.cc:1428] Setting key ed851998fe3cae10314dfd36328b7b81
I20260621 05:19:55.097805 20775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da69110180e4afaa46b01ac5ca19cb7, State: Initialized, Role: FOLLOWER
I20260621 05:19:55.097981 20775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da69110180e4afaa46b01ac5ca19cb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } }
I20260621 05:19:55.098074 20775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:55.098109 20775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:55.098160 20775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:55.098894 20775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da69110180e4afaa46b01ac5ca19cb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } }
I20260621 05:19:55.098999 20775 leader_election.cc:302] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1da69110180e4afaa46b01ac5ca19cb7; no voters:
I20260621 05:19:55.099165 20775 leader_election.cc:288] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:55.099206 20780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:55.099380 20780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [term 1 LEADER]: Becoming Leader. State: Replica: 1da69110180e4afaa46b01ac5ca19cb7, State: Running, Role: LEADER
I20260621 05:19:55.099406 20775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:55.099470 20780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da69110180e4afaa46b01ac5ca19cb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } }
I20260621 05:19:55.099822 20780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1da69110180e4afaa46b01ac5ca19cb7. Latest consensus state: current_term: 1 leader_uuid: "1da69110180e4afaa46b01ac5ca19cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da69110180e4afaa46b01ac5ca19cb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } } }
I20260621 05:19:55.099918 20780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:55.099833 20782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1da69110180e4afaa46b01ac5ca19cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da69110180e4afaa46b01ac5ca19cb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } } }
I20260621 05:19:55.100018 20782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:55.100791 20787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:55.101378 20787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:55.102927 20787 catalog_manager.cc:1347] Generated new cluster ID: 3a1bc4dc2fd74faf8a148385062938ac
I20260621 05:19:55.102984 20787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:55.104211 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:55.098226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57783 (local address 127.0.0.1:34751)
0621 05:19:55.098432 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:55.098441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:55.098474 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:19:55.098872 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:55.098882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:55.098944 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:55.099088 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:55.099350 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:55.100901 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:55.102020 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:55.102544 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:55.102648 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:55.103500 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:55.103503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:55.103504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:55.103760 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:55.103789 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:55.103799 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:55.103892 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:55.103980 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:19:55.104009 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
I20260621 05:19:55.105162 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:19:55.109148 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:55.106545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51817 (local address 127.0.0.1:34751)
0621 05:19:55.106581 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:55.106583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:55.106590 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:19:55.106609 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:55.106610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:55.106618 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:55.106670 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:55.106808 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:55.107286 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:55.108042 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:55.108268 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:55.108340 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:55.108918 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:55.108920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:55.108920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:55.108924 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:55.108936 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:55.108940 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:55.108991 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:55.109048 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 05:19:55.109059 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:19:55.119131 20787 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:55.119735 20787 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:55.123473 20787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1da69110180e4afaa46b01ac5ca19cb7: Generated new TSK 0
I20260621 05:19:55.123694 20787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:19:55.131121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 20707
W20260621 05:19:55.134886 20692 connection.cc:535] client connection to 127.0.0.1:34751 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34751): Connection reset by peer (error 104)
2026-06-21T05:19:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:55 dist-test-slave-czm5 krb5kdc[20806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:55 dist-test-slave-czm5 krb5kdc[20806](info): set up 2 sockets
Jun 21 05:19:55 dist-test-slave-czm5 krb5kdc[20806](info): commencing operation
krb5kdc: starting...
W20260621 05:19:56.159695 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:19:56 dist-test-slave-czm5 krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:19:56.274379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:56.354844 20819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:56.355103 20819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:56.355140 20819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:56.356333 20819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:56.356403 20819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:56.356428 20819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:56.356451 20819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:56.356495 20819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:56.357997 20819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:56.358278 20819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:56.358495 20819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:56.363238 20825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:56.363248 20827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:56.363369 20824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:56.363513 20819 server_base.cc:1034] running on GCE node
I20260621 05:19:56.363775 20819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:56.364099 20819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:56.365237 20819 hybrid_clock.cc:648] HybridClock initialized: now 1782019196365218 us; error 36 us; skew 500 ppm
Jun 21 05:19:56 dist-test-slave-czm5 krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:56.367517 20819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:19:56.367954 20819 webserver.cc:460] Webserver started at http://127.0.0.1:42211/ using document root <none> and password file <none>
I20260621 05:19:56.368141 20819 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:56.368180 20819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:56.368263 20819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:56.369254 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "7e082fba96434c37b15d0d6b7404ae1a"
format_stamp: "Formatted at 2026-06-21 05:19:56 on dist-test-slave-czm5"
server_key: "f1536239d42546c3ca96e4380fb237dc"
server_key_iv: "1ceb0008ea96dccb5860b90af065cd85"
server_key_version: "encryptionkey@0"
I20260621 05:19:56.369536 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e082fba96434c37b15d0d6b7404ae1a"
format_stamp: "Formatted at 2026-06-21 05:19:56 on dist-test-slave-czm5"
server_key: "f1536239d42546c3ca96e4380fb237dc"
server_key_iv: "1ceb0008ea96dccb5860b90af065cd85"
server_key_version: "encryptionkey@0"
I20260621 05:19:56.370745 20819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:19:56.371416 20834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:56.371709 20819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:19:56.371780 20819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "7e082fba96434c37b15d0d6b7404ae1a"
format_stamp: "Formatted at 2026-06-21 05:19:56 on dist-test-slave-czm5"
server_key: "f1536239d42546c3ca96e4380fb237dc"
server_key_iv: "1ceb0008ea96dccb5860b90af065cd85"
server_key_version: "encryptionkey@0"
I20260621 05:19:56.371873 20819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:56.374934 20819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:56.377367 20819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:56.377522 20819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:56.382180 20819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33635
I20260621 05:19:56.382241 20886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33635 every 8 connection(s)
I20260621 05:19:56.382537 20819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:56.383522 20887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:56.385684 20887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a: Bootstrap starting.
I20260621 05:19:56.386304 20887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:56.386597 20887 log.cc:824] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:56.387236 20887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a: No bootstrap required, opened a new log
I20260621 05:19:56.388850 20887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e082fba96434c37b15d0d6b7404ae1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33635 } }
I20260621 05:19:56.388990 20887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:56.389012 20887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e082fba96434c37b15d0d6b7404ae1a, State: Initialized, Role: FOLLOWER
I20260621 05:19:56.389220 20887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e082fba96434c37b15d0d6b7404ae1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33635 } }
I20260621 05:19:56.389262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 20819
I20260621 05:19:56.389304 20887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:56.389328 20887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:56.389379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:56.389390 20887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:56.389537 392 external_mini_cluster.cc:1428] Setting key db794813fe0f6ce9e0bcce1225981df6
I20260621 05:19:56.390076 20887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e082fba96434c37b15d0d6b7404ae1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33635 } }
I20260621 05:19:56.390211 20887 leader_election.cc:302] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e082fba96434c37b15d0d6b7404ae1a; no voters:
I20260621 05:19:56.390410 20887 leader_election.cc:288] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:56.390611 20892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:56.390691 20887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:56.390728 20892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [term 1 LEADER]: Becoming Leader. State: Replica: 7e082fba96434c37b15d0d6b7404ae1a, State: Running, Role: LEADER
I20260621 05:19:56.390846 20892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e082fba96434c37b15d0d6b7404ae1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33635 } }
I20260621 05:19:56.391253 20893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e082fba96434c37b15d0d6b7404ae1a. Latest consensus state: current_term: 1 leader_uuid: "7e082fba96434c37b15d0d6b7404ae1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e082fba96434c37b15d0d6b7404ae1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33635 } } }
I20260621 05:19:56.391266 20894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e082fba96434c37b15d0d6b7404ae1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e082fba96434c37b15d0d6b7404ae1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33635 } } }
I20260621 05:19:56.391533 20894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:56.391469 20893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:56.392393 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:56.390041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52851 (local address 127.0.0.1:33635)
0621 05:19:56.390603 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:56.390610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:56.390639 (+ 29us) server_negotiation.cc:407] Connection header received
0621 05:19:56.390795 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:56.390801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:56.390829 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:56.390973 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:56.391770 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:56.391774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:56.391774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:56.391779 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:56.391830 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:56.391848 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:56.392034 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:56.392118 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:19:56.392158 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":135,"threads_started":1}
W20260621 05:19:56.393020 20908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:19:56.393090 20908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:19:56.393177 20897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:56.393358 20897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:56.393518 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:19:56.394994 20897 catalog_manager.cc:1347] Generated new cluster ID: 6e009db96c454424a1516ee98a41bcb6
I20260621 05:19:56.395041 20897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:56.396178 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:56.395043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57191 (local address 127.0.0.1:33635)
0621 05:19:56.395076 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:56.395078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:56.395213 (+ 135us) server_negotiation.cc:407] Connection header received
0621 05:19:56.395283 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:56.395284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:56.395290 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:56.395343 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:56.395957 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:56.395958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:56.395958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:56.395962 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:56.395975 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:56.395979 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:56.396045 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:56.396094 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 05:19:56.396106 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:19:56.405371 20897 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:56.405534 20897 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:56.421978 20897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e082fba96434c37b15d0d6b7404ae1a: Generated new TSK 0
I20260621 05:19:56.422312 20897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:19:56.428182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 20819
2026-06-21T05:19:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:56 dist-test-slave-czm5 krb5kdc[20918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:56 dist-test-slave-czm5 krb5kdc[20918](info): set up 2 sockets
Jun 21 05:19:56 dist-test-slave-czm5 krb5kdc[20918](info): commencing operation
krb5kdc: starting...
W20260621 05:19:57.447681 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:19:57 dist-test-slave-czm5 krb5kdc[20918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:19:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:19:57.560439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:57.638406 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:57.638605 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:57.638628 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:57.639814 20931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:57.639854 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:57.639868 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:57.639881 20931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:57.639936 20931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:57.641378 20931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:57.641580 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:57.641809 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:57.646663 20936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:57.646718 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:57.646704 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:57.646939 20931 server_base.cc:1034] running on GCE node
I20260621 05:19:57.647189 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:57.647482 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:57.648643 20931 hybrid_clock.cc:648] HybridClock initialized: now 1782019197648625 us; error 47 us; skew 500 ppm
Jun 21 05:19:57 dist-test-slave-czm5 krb5kdc[20918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:57.651088 20931 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:19:57.651455 20931 webserver.cc:460] Webserver started at http://127.0.0.1:40913/ using document root <none> and password file <none>
I20260621 05:19:57.651716 20931 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:57.651762 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:57.651849 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:57.652835 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "affe6d42cde94783992873ab2024e9d4"
format_stamp: "Formatted at 2026-06-21 05:19:57 on dist-test-slave-czm5"
server_key: "a4b317c653c3a56560251e32d99cd9fc"
server_key_iv: "5f1fa391526eed8aa03c5574a4c97db2"
server_key_version: "encryptionkey@0"
I20260621 05:19:57.653139 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "affe6d42cde94783992873ab2024e9d4"
format_stamp: "Formatted at 2026-06-21 05:19:57 on dist-test-slave-czm5"
server_key: "a4b317c653c3a56560251e32d99cd9fc"
server_key_iv: "5f1fa391526eed8aa03c5574a4c97db2"
server_key_version: "encryptionkey@0"
I20260621 05:19:57.654328 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:19:57.655021 20946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:57.655179 20931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:57.655248 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "affe6d42cde94783992873ab2024e9d4"
format_stamp: "Formatted at 2026-06-21 05:19:57 on dist-test-slave-czm5"
server_key: "a4b317c653c3a56560251e32d99cd9fc"
server_key_iv: "5f1fa391526eed8aa03c5574a4c97db2"
server_key_version: "encryptionkey@0"
I20260621 05:19:57.655299 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:57.671731 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:57.673453 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:57.673580 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:57.677659 20931 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42695
I20260621 05:19:57.677985 20998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42695 every 8 connection(s)
I20260621 05:19:57.678051 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:57.678761 20999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:57.680990 20999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4: Bootstrap starting.
I20260621 05:19:57.681591 20999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:57.681869 20999 log.cc:824] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:57.682531 20999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4: No bootstrap required, opened a new log
I20260621 05:19:57.684029 20999 raft_consensus.cc:348] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affe6d42cde94783992873ab2024e9d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260621 05:19:57.684173 20999 raft_consensus.cc:374] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:57.684219 20999 raft_consensus.cc:729] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affe6d42cde94783992873ab2024e9d4, State: Initialized, Role: FOLLOWER
I20260621 05:19:57.684371 20999 consensus_queue.cc:260] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affe6d42cde94783992873ab2024e9d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260621 05:19:57.684459 20999 raft_consensus.cc:388] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:57.684504 20999 raft_consensus.cc:482] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:57.684556 20999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:57.685034 20999 raft_consensus.cc:504] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affe6d42cde94783992873ab2024e9d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260621 05:19:57.685113 20999 leader_election.cc:302] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: affe6d42cde94783992873ab2024e9d4; no voters:
I20260621 05:19:57.685294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 20931
I20260621 05:19:57.685316 20999 leader_election.cc:288] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:57.685348 21002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:57.685422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:57.685462 21002 raft_consensus.cc:686] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [term 1 LEADER]: Becoming Leader. State: Replica: affe6d42cde94783992873ab2024e9d4, State: Running, Role: LEADER
I20260621 05:19:57.685541 21002 consensus_queue.cc:237] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affe6d42cde94783992873ab2024e9d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260621 05:19:57.685577 20999 sys_catalog.cc:564] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:57.685621 392 external_mini_cluster.cc:1428] Setting key 8e993dec79e98f4f4a0f3418f3b6f3d6
I20260621 05:19:57.685832 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "affe6d42cde94783992873ab2024e9d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affe6d42cde94783992873ab2024e9d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } } }
I20260621 05:19:57.685983 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:57.686168 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader affe6d42cde94783992873ab2024e9d4. Latest consensus state: current_term: 1 leader_uuid: "affe6d42cde94783992873ab2024e9d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affe6d42cde94783992873ab2024e9d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } } }
I20260621 05:19:57.686285 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:57.686666 21010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:57.687177 21010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:57.688707 21010 catalog_manager.cc:1347] Generated new cluster ID: 7bfde111a36d4b8987eb31e361fd2a62
I20260621 05:19:57.688758 21010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:19:57 dist-test-slave-czm5 krb5kdc[20918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:19:57.695834 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:57.686366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59257 (local address 127.0.0.1:42695)
0621 05:19:57.686561 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:57.686569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:57.686599 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:19:57.686707 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:57.686713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:57.686765 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:57.686929 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:57.687794 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:57.688998 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:57.689728 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:57.690145 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:57.690261 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:57.692609 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:57.692613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:57.692614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:57.692625 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:57.694210 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:57.694566 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:57.694569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:57.694569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:57.694661 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:57.694886 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:57.694887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:57.694887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:57.695131 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:57.695321 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:57.695454 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 05:19:57.695598 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260621 05:19:57.696789 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:19:57.697245 21010 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:57.697899 21010 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:57.703783 21010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P affe6d42cde94783992873ab2024e9d4: Generated new TSK 0
I20260621 05:19:57.703991 21010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:19:57.705751 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:57.698247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40537 (local address 127.0.0.1:42695)
0621 05:19:57.698282 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:57.698284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:57.699812 (+ 1528us) server_negotiation.cc:407] Connection header received
0621 05:19:57.699886 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:57.699887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:57.699897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:57.699939 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:57.700459 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:57.700962 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:57.701699 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:57.701918 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:57.702000 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:57.703133 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:57.703135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:57.703135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:19:57.703142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:57.704616 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:57.704992 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:57.704993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:57.704993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:57.705045 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:19:57.705254 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:19:57.705255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:19:57.705255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:19:57.705360 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:57.705459 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:57.705559 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 05:19:57.705653 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260621 05:19:57.707669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 20931
2026-06-21T05:19:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:57 dist-test-slave-czm5 krb5kdc[21030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:57 dist-test-slave-czm5 krb5kdc[21030](info): set up 2 sockets
Jun 21 05:19:57 dist-test-slave-czm5 krb5kdc[21030](info): commencing operation
krb5kdc: starting...
W20260621 05:19:58.727828 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:19:58 dist-test-slave-czm5 krb5kdc[21030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:19:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:19:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:19:58.836380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:19:58.913197 21043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:19:58.913406 21043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:19:58.913442 21043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:19:58.914536 21043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:19:58.914615 21043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:19:58.914656 21043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:19:58.914683 21043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:19:58.914712 21043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:19:58.916106 21043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:19:58.916347 21043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:19:58.916546 21043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:19:58.921068 21049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:58.921116 21048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:19:58.921082 21051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:19:58.921373 21043 server_base.cc:1034] running on GCE node
I20260621 05:19:58.921617 21043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:19:58.921871 21043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:19:58.923031 21043 hybrid_clock.cc:648] HybridClock initialized: now 1782019198923018 us; error 40 us; skew 500 ppm
Jun 21 05:19:58 dist-test-slave-czm5 krb5kdc[21030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:19:58.925340 21043 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:19:58.925714 21043 webserver.cc:460] Webserver started at http://127.0.0.1:42637/ using document root <none> and password file <none>
I20260621 05:19:58.925925 21043 fs_manager.cc:359] Metadata directory not provided
I20260621 05:19:58.925987 21043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:19:58.926121 21043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:19:58.927104 21043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "571f9cf20a394e54a756834279da9916"
format_stamp: "Formatted at 2026-06-21 05:19:58 on dist-test-slave-czm5"
server_key: "4d6b8674c51f369daf4b832f17e180f9"
server_key_iv: "95f6d52052208b205c4efb7779d9593b"
server_key_version: "encryptionkey@0"
I20260621 05:19:58.927454 21043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "571f9cf20a394e54a756834279da9916"
format_stamp: "Formatted at 2026-06-21 05:19:58 on dist-test-slave-czm5"
server_key: "4d6b8674c51f369daf4b832f17e180f9"
server_key_iv: "95f6d52052208b205c4efb7779d9593b"
server_key_version: "encryptionkey@0"
I20260621 05:19:58.928743 21043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:19:58.929421 21058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:19:58.929605 21043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:19:58.929695 21043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "571f9cf20a394e54a756834279da9916"
format_stamp: "Formatted at 2026-06-21 05:19:58 on dist-test-slave-czm5"
server_key: "4d6b8674c51f369daf4b832f17e180f9"
server_key_iv: "95f6d52052208b205c4efb7779d9593b"
server_key_version: "encryptionkey@0"
I20260621 05:19:58.929792 21043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:19:58.943502 21043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:19:58.945240 21043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:19:58.945418 21043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:19:58.949270 21043 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43443
I20260621 05:19:58.949312 21110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43443 every 8 connection(s)
I20260621 05:19:58.949671 21043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:19:58.950389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21043
I20260621 05:19:58.950510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:19:58.950670 21111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:19:58.950726 392 external_mini_cluster.cc:1428] Setting key 6741ac5eef351cb78561a9053dcbaad3
I20260621 05:19:58.953570 21111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916: Bootstrap starting.
I20260621 05:19:58.954216 21111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916: Neither blocks nor log segments found. Creating new log.
I20260621 05:19:58.954502 21111 log.cc:824] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916: Log is configured to *not* fsync() on all Append() calls
I20260621 05:19:58.955212 21111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916: No bootstrap required, opened a new log
I20260621 05:19:58.956219 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:58.951269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50829 (local address 127.0.0.1:43443)
0621 05:19:58.951430 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:58.951439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:58.951476 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:19:58.952043 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:58.952049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:58.952095 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:58.952205 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:58.952452 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:58.953299 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:58.954368 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:58.954803 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:58.954884 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:58.955574 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:58.955578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:58.955578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:58.955783 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:58.955816 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:58.955829 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:58.955894 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:58.955955 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 05:19:58.956013 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260621 05:19:58.957394 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:19:58.957552 21111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571f9cf20a394e54a756834279da9916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43443 } }
I20260621 05:19:58.957688 21111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:19:58.957711 21111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571f9cf20a394e54a756834279da9916, State: Initialized, Role: FOLLOWER
I20260621 05:19:58.957844 21111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571f9cf20a394e54a756834279da9916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43443 } }
I20260621 05:19:58.957922 21111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:19:58.957962 21111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:19:58.958021 21111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:19:58.958551 21111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571f9cf20a394e54a756834279da9916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43443 } }
I20260621 05:19:58.958657 21111 leader_election.cc:302] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 571f9cf20a394e54a756834279da9916; no voters:
I20260621 05:19:58.958828 21111 leader_election.cc:288] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:19:58.958861 21120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:19:58.959033 21120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [term 1 LEADER]: Becoming Leader. State: Replica: 571f9cf20a394e54a756834279da9916, State: Running, Role: LEADER
I20260621 05:19:58.959125 21120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571f9cf20a394e54a756834279da9916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43443 } }
I20260621 05:19:58.959132 21111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:19:58.959429 21120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 571f9cf20a394e54a756834279da9916. Latest consensus state: current_term: 1 leader_uuid: "571f9cf20a394e54a756834279da9916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571f9cf20a394e54a756834279da9916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43443 } } }
I20260621 05:19:58.959453 21123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "571f9cf20a394e54a756834279da9916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571f9cf20a394e54a756834279da9916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43443 } } }
I20260621 05:19:58.959517 21120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:58.959525 21123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916 [sys.catalog]: This master's current role is: LEADER
I20260621 05:19:58.959756 21126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:19:58.960242 21126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:19:58.961810 21126 catalog_manager.cc:1347] Generated new cluster ID: 7290838177d04d5b9b7ba282380e2523
I20260621 05:19:58.961864 21126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:19:58.963728 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:19:58.958811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57959 (local address 127.0.0.1:43443)
0621 05:19:58.958848 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:19:58.958850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:19:58.960921 (+ 2071us) server_negotiation.cc:407] Connection header received
0621 05:19:58.961059 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:19:58.961060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:19:58.961071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:19:58.961139 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:19:58.961268 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:58.961941 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:58.962600 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:19:58.962802 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:19:58.962883 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:19:58.963463 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:19:58.963464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:19:58.963465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:19:58.963469 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:19:58.963481 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:19:58.963484 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:19:58.963525 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0621 05:19:58.963600 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:19:58.963638 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:19:58.977846 21126 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:19:58.978551 21126 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:19:58.988917 21126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 571f9cf20a394e54a756834279da9916: Generated new TSK 0
I20260621 05:19:58.989089 21126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:19:58.998080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21043
2026-06-21T05:19:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:19:59 dist-test-slave-czm5 krb5kdc[21142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:19:59 dist-test-slave-czm5 krb5kdc[21142](info): set up 2 sockets
Jun 21 05:19:59 dist-test-slave-czm5 krb5kdc[21142](info): commencing operation
krb5kdc: starting...
W20260621 05:20:00.011458 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:00 dist-test-slave-czm5 krb5kdc[21142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:20:00.122932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:00.200865 21155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:00.201090 21155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:00.201129 21155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:00.202265 21155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:00.202329 21155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:00.202354 21155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:00.202396 21155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:00.202419 21155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:00.204044 21155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:00.204296 21155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:00.204497 21155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:00.208976 21161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:00.209026 21160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:00.209026 21163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:00.209415 21155 server_base.cc:1034] running on GCE node
I20260621 05:20:00.209661 21155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:00.209900 21155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:00.211035 21155 hybrid_clock.cc:648] HybridClock initialized: now 1782019200211010 us; error 56 us; skew 500 ppm
Jun 21 05:20:00 dist-test-slave-czm5 krb5kdc[21142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:00.217554 21155 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:20:00.217989 21155 webserver.cc:460] Webserver started at http://127.0.0.1:45735/ using document root <none> and password file <none>
I20260621 05:20:00.218215 21155 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:00.218283 21155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:00.218417 21155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:00.219434 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "b3c5b599dc8047e99af745a330e6701f"
format_stamp: "Formatted at 2026-06-21 05:20:00 on dist-test-slave-czm5"
server_key: "6a9c22c3ee6bfabc98f38a834ea9f55d"
server_key_iv: "0f2934c939bef965d3a167cbcf16b184"
server_key_version: "encryptionkey@0"
I20260621 05:20:00.219811 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3c5b599dc8047e99af745a330e6701f"
format_stamp: "Formatted at 2026-06-21 05:20:00 on dist-test-slave-czm5"
server_key: "6a9c22c3ee6bfabc98f38a834ea9f55d"
server_key_iv: "0f2934c939bef965d3a167cbcf16b184"
server_key_version: "encryptionkey@0"
I20260621 05:20:00.221104 21155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:00.221810 21170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:00.221982 21155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:20:00.222078 21155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "b3c5b599dc8047e99af745a330e6701f"
format_stamp: "Formatted at 2026-06-21 05:20:00 on dist-test-slave-czm5"
server_key: "6a9c22c3ee6bfabc98f38a834ea9f55d"
server_key_iv: "0f2934c939bef965d3a167cbcf16b184"
server_key_version: "encryptionkey@0"
I20260621 05:20:00.222160 21155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:00.225455 21155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:00.227156 21155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:00.227319 21155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:00.231038 21155 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42367
I20260621 05:20:00.231067 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42367 every 8 connection(s)
I20260621 05:20:00.231369 21155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:00.232328 21223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:00.234467 21223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f: Bootstrap starting.
I20260621 05:20:00.234952 21223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:00.235170 21223 log.cc:824] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:00.235877 21223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f: No bootstrap required, opened a new log
I20260621 05:20:00.237121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21155
I20260621 05:20:00.237251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:00.237427 392 external_mini_cluster.cc:1428] Setting key 40b608e9c441d096b2d9a0a96483df77
I20260621 05:20:00.237789 21223 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c5b599dc8047e99af745a330e6701f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } }
I20260621 05:20:00.237963 21223 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:00.238003 21223 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c5b599dc8047e99af745a330e6701f, State: Initialized, Role: FOLLOWER
I20260621 05:20:00.238184 21223 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c5b599dc8047e99af745a330e6701f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } }
I20260621 05:20:00.238270 21223 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:00.238300 21223 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:00.238345 21223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:00.239080 21223 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c5b599dc8047e99af745a330e6701f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } }
I20260621 05:20:00.239189 21223 leader_election.cc:302] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3c5b599dc8047e99af745a330e6701f; no voters:
I20260621 05:20:00.239356 21223 leader_election.cc:288] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:00.239392 21228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:00.239559 21228 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [term 1 LEADER]: Becoming Leader. State: Replica: b3c5b599dc8047e99af745a330e6701f, State: Running, Role: LEADER
I20260621 05:20:00.239593 21223 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:00.239687 21228 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c5b599dc8047e99af745a330e6701f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } }
I20260621 05:20:00.240064 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3c5b599dc8047e99af745a330e6701f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c5b599dc8047e99af745a330e6701f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } } }
I20260621 05:20:00.240070 21229 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3c5b599dc8047e99af745a330e6701f. Latest consensus state: current_term: 1 leader_uuid: "b3c5b599dc8047e99af745a330e6701f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c5b599dc8047e99af745a330e6701f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } } }
I20260621 05:20:00.240161 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:00.240182 21229 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:00.240362 21236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:00.240883 21236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:00.239579 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:00.237910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38647 (local address 127.0.0.1:42367)
0621 05:20:00.238128 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:00.238136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:00.238167 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:20:00.238272 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:00.238279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:00.238310 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:00.238444 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:00.239023 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:00.239026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:00.239027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:00.239030 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:00.239067 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:00.239081 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:00.239161 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:00.239283 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 05:20:00.239328 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260621 05:20:00.242395 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:00.242477 21236 catalog_manager.cc:1347] Generated new cluster ID: 5ae0448986d846a4b3271c8a67fd3de1
I20260621 05:20:00.242528 21236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:00.245033 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:00.243915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56213 (local address 127.0.0.1:42367)
0621 05:20:00.243950 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:00.243952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:00.244180 (+ 228us) server_negotiation.cc:407] Connection header received
0621 05:20:00.244235 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:00.244236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:00.244241 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:00.244294 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:00.244836 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:00.244837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:00.244837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:00.244841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:00.244854 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:00.244857 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:00.244901 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:00.244959 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 05:20:00.244973 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:20:00.249126 21236 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:00.249266 21236 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:00.254094 21236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3c5b599dc8047e99af745a330e6701f: Generated new TSK 0
I20260621 05:20:00.254249 21236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:00.259990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21155
2026-06-21T05:20:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:00 dist-test-slave-czm5 krb5kdc[21254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:00 dist-test-slave-czm5 krb5kdc[21254](info): set up 2 sockets
Jun 21 05:20:00 dist-test-slave-czm5 krb5kdc[21254](info): commencing operation
krb5kdc: starting...
W20260621 05:20:01.275431 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:01 dist-test-slave-czm5 krb5kdc[21254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:20:01.384495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:01.462628 21267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:01.462822 21267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:01.462842 21267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:01.463887 21267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:01.463919 21267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:01.463933 21267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:01.463945 21267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:01.463956 21267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:01.465271 21267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:01.465442 21267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:01.465596 21267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:01.470134 21273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:01.470242 21275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:01.470170 21272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:01.470340 21267 server_base.cc:1034] running on GCE node
I20260621 05:20:01.470588 21267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:01.470845 21267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:01.471976 21267 hybrid_clock.cc:648] HybridClock initialized: now 1782019201471964 us; error 25 us; skew 500 ppm
Jun 21 05:20:01 dist-test-slave-czm5 krb5kdc[21254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:01.474295 21267 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:20:01.474735 21267 webserver.cc:460] Webserver started at http://127.0.0.1:46759/ using document root <none> and password file <none>
I20260621 05:20:01.474956 21267 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:01.475025 21267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:01.475184 21267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:01.476262 21267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5"
format_stamp: "Formatted at 2026-06-21 05:20:01 on dist-test-slave-czm5"
server_key: "addcd87141ee446cd074f6dda16a8b3c"
server_key_iv: "1d05547e1a1f95d0043a80c7d709e798"
server_key_version: "encryptionkey@0"
I20260621 05:20:01.476624 21267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5"
format_stamp: "Formatted at 2026-06-21 05:20:01 on dist-test-slave-czm5"
server_key: "addcd87141ee446cd074f6dda16a8b3c"
server_key_iv: "1d05547e1a1f95d0043a80c7d709e798"
server_key_version: "encryptionkey@0"
I20260621 05:20:01.477913 21267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:01.478605 21282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:01.478765 21267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:01.478866 21267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5"
format_stamp: "Formatted at 2026-06-21 05:20:01 on dist-test-slave-czm5"
server_key: "addcd87141ee446cd074f6dda16a8b3c"
server_key_iv: "1d05547e1a1f95d0043a80c7d709e798"
server_key_version: "encryptionkey@0"
I20260621 05:20:01.478937 21267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:01.505800 21267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:01.507592 21267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:01.507774 21267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:01.511905 21267 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36923
I20260621 05:20:01.511955 21334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36923 every 8 connection(s)
I20260621 05:20:01.512321 21267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:01.513134 21335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:01.515218 21335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5: Bootstrap starting.
I20260621 05:20:01.515764 21335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:01.516028 21335 log.cc:824] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:01.516637 21335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5: No bootstrap required, opened a new log
I20260621 05:20:01.518237 21335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260621 05:20:01.518368 21335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:01.518389 21335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a9a1f697304dd5bcc5ba0e123ff6f5, State: Initialized, Role: FOLLOWER
I20260621 05:20:01.518579 21335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260621 05:20:01.518638 21335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:01.518658 21335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:01.518705 21335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:01.518942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21267
I20260621 05:20:01.519079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:01.519263 392 external_mini_cluster.cc:1428] Setting key 87f6f25b6bc46e46fa5edcf78b40a116
I20260621 05:20:01.519271 21335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260621 05:20:01.519366 21335 leader_election.cc:302] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33a9a1f697304dd5bcc5ba0e123ff6f5; no voters:
I20260621 05:20:01.519533 21335 leader_election.cc:288] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:01.519569 21338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:01.519731 21338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [term 1 LEADER]: Becoming Leader. State: Replica: 33a9a1f697304dd5bcc5ba0e123ff6f5, State: Running, Role: LEADER
I20260621 05:20:01.519853 21335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:01.519860 21338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260621 05:20:01.520246 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33a9a1f697304dd5bcc5ba0e123ff6f5. Latest consensus state: current_term: 1 leader_uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } } }
I20260621 05:20:01.520305 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:01.520406 21340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a9a1f697304dd5bcc5ba0e123ff6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } } }
I20260621 05:20:01.520483 21340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:01.520593 21344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:01.521159 21344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:01.522828 21344 catalog_manager.cc:1347] Generated new cluster ID: fade77c50d464825bbfe406cfd8b1577
I20260621 05:20:01.522899 21344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:20:01 dist-test-slave-czm5 krb5kdc[21254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:20:01.529472 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:01.519761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:41839 (local address 127.0.0.1:36923)
0621 05:20:01.520041 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:01.520049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:01.520081 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:20:01.520193 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:01.520201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:01.520257 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:01.521439 (+ 1182us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:01.521554 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:01.522739 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:01.523541 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:01.524002 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:01.524117 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:01.526261 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:01.526265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:01.526265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:01.526276 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:01.527952 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:01.528330 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:01.528333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:01.528333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:01.528427 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:01.528661 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:01.528662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:01.528662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:01.528886 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:01.529085 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:01.529163 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:20:01.529291 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":64,"threads_started":1}
I20260621 05:20:01.530324 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:01.537422 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:01.531614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:40175 (local address 127.0.0.1:36923)
0621 05:20:01.531664 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:01.531666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:01.531779 (+ 113us) server_negotiation.cc:407] Connection header received
0621 05:20:01.531861 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:01.531862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:01.531870 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:01.531913 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:01.532389 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:01.532836 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:01.533481 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:01.533708 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:01.533781 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:01.534863 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:01.534865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:01.534865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:01.534872 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:01.536265 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:01.536624 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:01.536625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:01.536625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:01.536677 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:01.536909 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:01.536910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:01.536910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:01.537018 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:01.537135 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:01.537243 (+ 108us) server_negotiation.cc:299] Negotiation successful
0621 05:20:01.537310 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:20:01.545127 21344 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:01.545766 21344 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:01.558629 21344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33a9a1f697304dd5bcc5ba0e123ff6f5: Generated new TSK 0
I20260621 05:20:01.558795 21344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:01.568928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21267
W20260621 05:20:01.573325 21252 connection.cc:535] client connection to 127.0.0.1:36923 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36923): Connection reset by peer (error 104)
2026-06-21T05:20:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:01 dist-test-slave-czm5 krb5kdc[21366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:01 dist-test-slave-czm5 krb5kdc[21366](info): set up 2 sockets
Jun 21 05:20:01 dist-test-slave-czm5 krb5kdc[21366](info): commencing operation
krb5kdc: starting...
W20260621 05:20:02.587606 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:02 dist-test-slave-czm5 krb5kdc[21366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:20:02.700541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:02.782133 21379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:02.782394 21379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:02.782433 21379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:02.783612 21379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:02.783706 21379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:02.783730 21379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:02.783762 21379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:02.783788 21379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:02.785249 21379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:02.785535 21379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:02.785766 21379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:02.790704 21387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:02.790781 21384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:02.790814 21379 server_base.cc:1034] running on GCE node
W20260621 05:20:02.790787 21385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:02.791137 21379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:02.791467 21379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:02.792629 21379 hybrid_clock.cc:648] HybridClock initialized: now 1782019202792616 us; error 44 us; skew 500 ppm
Jun 21 05:20:02 dist-test-slave-czm5 krb5kdc[21366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:02.795053 21379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:20:02.795480 21379 webserver.cc:460] Webserver started at http://127.0.0.1:37033/ using document root <none> and password file <none>
I20260621 05:20:02.795750 21379 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:02.795818 21379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:02.795984 21379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:02.797045 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "94719fcaf411440ea2eac9f97b08bc42"
format_stamp: "Formatted at 2026-06-21 05:20:02 on dist-test-slave-czm5"
server_key: "bae79cdd585e31621723f8e95574ae17"
server_key_iv: "03e642e4215dbd90b1c5b74a4bd5ea27"
server_key_version: "encryptionkey@0"
I20260621 05:20:02.797408 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "94719fcaf411440ea2eac9f97b08bc42"
format_stamp: "Formatted at 2026-06-21 05:20:02 on dist-test-slave-czm5"
server_key: "bae79cdd585e31621723f8e95574ae17"
server_key_iv: "03e642e4215dbd90b1c5b74a4bd5ea27"
server_key_version: "encryptionkey@0"
I20260621 05:20:02.798704 21379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:02.799403 21394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:02.799576 21379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:02.799693 21379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "94719fcaf411440ea2eac9f97b08bc42"
format_stamp: "Formatted at 2026-06-21 05:20:02 on dist-test-slave-czm5"
server_key: "bae79cdd585e31621723f8e95574ae17"
server_key_iv: "03e642e4215dbd90b1c5b74a4bd5ea27"
server_key_version: "encryptionkey@0"
I20260621 05:20:02.799777 21379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:02.810523 21379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:02.812361 21379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:02.812523 21379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:02.816359 21379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33341
I20260621 05:20:02.816403 21446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33341 every 8 connection(s)
I20260621 05:20:02.816726 21379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:02.817627 21447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:02.819905 21447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42: Bootstrap starting.
I20260621 05:20:02.820462 21447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:02.820739 21447 log.cc:824] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:02.821444 21447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42: No bootstrap required, opened a new log
I20260621 05:20:02.823002 21447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94719fcaf411440ea2eac9f97b08bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } }
I20260621 05:20:02.823148 21447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:02.823197 21447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94719fcaf411440ea2eac9f97b08bc42, State: Initialized, Role: FOLLOWER
I20260621 05:20:02.823355 21447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94719fcaf411440ea2eac9f97b08bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } }
I20260621 05:20:02.823438 21447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:02.823491 21447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:02.823544 21447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:02.824077 21447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94719fcaf411440ea2eac9f97b08bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } }
I20260621 05:20:02.824187 21447 leader_election.cc:302] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94719fcaf411440ea2eac9f97b08bc42; no voters:
I20260621 05:20:02.824370 21447 leader_election.cc:288] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:02.824410 21450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:02.824590 21450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [term 1 LEADER]: Becoming Leader. State: Replica: 94719fcaf411440ea2eac9f97b08bc42, State: Running, Role: LEADER
I20260621 05:20:02.824633 21447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:02.824717 21450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94719fcaf411440ea2eac9f97b08bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } }
I20260621 05:20:02.824991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21379
I20260621 05:20:02.825101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:02.825304 392 external_mini_cluster.cc:1428] Setting key 90cdb6f772741b483d09d2c37f5e843d
I20260621 05:20:02.825891 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94719fcaf411440ea2eac9f97b08bc42. Latest consensus state: current_term: 1 leader_uuid: "94719fcaf411440ea2eac9f97b08bc42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94719fcaf411440ea2eac9f97b08bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } } }
I20260621 05:20:02.826206 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:02.825970 21451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94719fcaf411440ea2eac9f97b08bc42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94719fcaf411440ea2eac9f97b08bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } } }
I20260621 05:20:02.826432 21451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42 [sys.catalog]: This master's current role is: LEADER
W20260621 05:20:02.826472 21466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:02.826538 21466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:02.826915 21468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:02.827091 21468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:02.828627 21468 catalog_manager.cc:1347] Generated new cluster ID: 07a2d619b9ba46feaa59ad08cb699b9d
I20260621 05:20:02.828680 21468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:02.831519 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:02.825778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:41371 (local address 127.0.0.1:33341)
0621 05:20:02.826118 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:02.826127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:02.826777 (+ 650us) server_negotiation.cc:407] Connection header received
0621 05:20:02.827012 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:02.827020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:02.827083 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:02.827219 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:02.827469 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:02.828797 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:02.829595 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:02.830014 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:02.830133 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:02.830878 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:02.830882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:02.830882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:02.831060 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:02.831090 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:02.831101 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:02.831193 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:02.831272 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 05:20:02.831302 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":55,"threads_started":1}
I20260621 05:20:02.832460 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:02.836859 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:02.833887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:52929 (local address 127.0.0.1:33341)
0621 05:20:02.833923 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:02.833925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:02.834043 (+ 118us) server_negotiation.cc:407] Connection header received
0621 05:20:02.834114 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:02.834115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:02.834126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:02.834182 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:02.834308 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:02.834791 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:02.835501 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:02.835833 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:02.835921 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:02.836588 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:02.836590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:02.836590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:02.836595 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:02.836607 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:02.836611 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:02.836662 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:02.836725 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 05:20:02.836738 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:20:02.840233 21468 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:02.840737 21468 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:02.845562 21468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94719fcaf411440ea2eac9f97b08bc42: Generated new TSK 0
I20260621 05:20:02.845726 21468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:02.848829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21379
W20260621 05:20:02.852843 21364 connection.cc:535] client connection to 127.0.0.1:33341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33341): Connection reset by peer (error 104)
2026-06-21T05:20:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:02 dist-test-slave-czm5 krb5kdc[21478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:02 dist-test-slave-czm5 krb5kdc[21478](info): set up 2 sockets
Jun 21 05:20:02 dist-test-slave-czm5 krb5kdc[21478](info): commencing operation
krb5kdc: starting...
W20260621 05:20:03.863559 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:03 dist-test-slave-czm5 krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:20:03.976358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:04.055230 21491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:04.055462 21491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:04.055500 21491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:04.056695 21491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:04.056767 21491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:04.056793 21491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:04.056845 21491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:04.056869 21491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:04.058303 21491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:04.058573 21491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:04.058804 21491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:04.063680 21497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:04.063728 21496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:04.063687 21499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:04.063828 21491 server_base.cc:1034] running on GCE node
I20260621 05:20:04.064072 21491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:04.064357 21491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:04.065523 21491 hybrid_clock.cc:648] HybridClock initialized: now 1782019204065500 us; error 35 us; skew 500 ppm
Jun 21 05:20:04 dist-test-slave-czm5 krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:04.067914 21491 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:20:04.068392 21491 webserver.cc:460] Webserver started at http://127.0.0.1:37031/ using document root <none> and password file <none>
I20260621 05:20:04.068631 21491 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:04.068702 21491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:04.068858 21491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:04.069873 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "4935d2f045e14b25b18c8fdb958a7b04"
format_stamp: "Formatted at 2026-06-21 05:20:04 on dist-test-slave-czm5"
server_key: "d60bbf7a15c47605ba74e9d41c947040"
server_key_iv: "7996b3994363b50013782dd8ffbc57a4"
server_key_version: "encryptionkey@0"
I20260621 05:20:04.070235 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "4935d2f045e14b25b18c8fdb958a7b04"
format_stamp: "Formatted at 2026-06-21 05:20:04 on dist-test-slave-czm5"
server_key: "d60bbf7a15c47605ba74e9d41c947040"
server_key_iv: "7996b3994363b50013782dd8ffbc57a4"
server_key_version: "encryptionkey@0"
I20260621 05:20:04.071519 21491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:04.072263 21506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:04.072430 21491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:20:04.072530 21491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "4935d2f045e14b25b18c8fdb958a7b04"
format_stamp: "Formatted at 2026-06-21 05:20:04 on dist-test-slave-czm5"
server_key: "d60bbf7a15c47605ba74e9d41c947040"
server_key_iv: "7996b3994363b50013782dd8ffbc57a4"
server_key_version: "encryptionkey@0"
I20260621 05:20:04.072613 21491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:04.075789 21491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:04.077617 21491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:04.077775 21491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:04.081869 21491 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42963
I20260621 05:20:04.081897 21558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42963 every 8 connection(s)
I20260621 05:20:04.082238 21491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:04.083156 21559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:04.085373 21559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04: Bootstrap starting.
I20260621 05:20:04.085937 21559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:04.086217 21559 log.cc:824] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:04.086851 21559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04: No bootstrap required, opened a new log
I20260621 05:20:04.088488 21559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935d2f045e14b25b18c8fdb958a7b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260621 05:20:04.088619 21559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:04.088640 21559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4935d2f045e14b25b18c8fdb958a7b04, State: Initialized, Role: FOLLOWER
I20260621 05:20:04.088814 21559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935d2f045e14b25b18c8fdb958a7b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260621 05:20:04.088877 21559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:04.088919 21559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:04.088968 21559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:04.089506 21559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935d2f045e14b25b18c8fdb958a7b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260621 05:20:04.089614 21559 leader_election.cc:302] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4935d2f045e14b25b18c8fdb958a7b04; no voters:
I20260621 05:20:04.089788 21559 leader_election.cc:288] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:04.089807 21562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:04.089955 21562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [term 1 LEADER]: Becoming Leader. State: Replica: 4935d2f045e14b25b18c8fdb958a7b04, State: Running, Role: LEADER
I20260621 05:20:04.090040 21559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:04.090085 21562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935d2f045e14b25b18c8fdb958a7b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260621 05:20:04.090544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21491
I20260621 05:20:04.090685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:04.090862 392 external_mini_cluster.cc:1428] Setting key fc2195503fee5c2f905ec3fe36be5a6a
I20260621 05:20:04.091920 21564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4935d2f045e14b25b18c8fdb958a7b04. Latest consensus state: current_term: 1 leader_uuid: "4935d2f045e14b25b18c8fdb958a7b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935d2f045e14b25b18c8fdb958a7b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } } }
I20260621 05:20:04.092137 21564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [sys.catalog]: This master's current role is: LEADER
W20260621 05:20:04.092296 21579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:04.092434 21579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:04.092295 21563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4935d2f045e14b25b18c8fdb958a7b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935d2f045e14b25b18c8fdb958a7b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } } }
I20260621 05:20:04.092579 21563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:04.092655 21581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:04.092877 21581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:04.093209 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:04.091386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:37065 (local address 127.0.0.1:42963)
0621 05:20:04.091669 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:04.091678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:04.091710 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:20:04.091829 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:04.091835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:04.091868 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:04.092271 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:04.092745 (+ 474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:04.092749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:04.092750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:04.092754 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:04.092803 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:04.092817 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:04.092902 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:04.092995 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 05:20:04.093040 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260621 05:20:04.094133 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:04.095978 21581 catalog_manager.cc:1347] Generated new cluster ID: f8402a36b77945238868d754a07909b8
I20260621 05:20:04.096041 21581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:04.097131 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:04.095467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:44843 (local address 127.0.0.1:42963)
0621 05:20:04.095499 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:04.095500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:04.096130 (+ 630us) server_negotiation.cc:407] Connection header received
0621 05:20:04.096229 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:04.096230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:04.096234 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:04.096305 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:04.096909 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:04.096911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:04.096911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:04.096915 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:04.096930 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:04.096934 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:04.096996 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:04.097053 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 05:20:04.097066 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:20:04.112399 21581 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:04.112584 21581 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:04.117660 21581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4935d2f045e14b25b18c8fdb958a7b04: Generated new TSK 0
I20260621 05:20:04.117836 21581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:04.120944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21491
2026-06-21T05:20:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:04 dist-test-slave-czm5 krb5kdc[21590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:04 dist-test-slave-czm5 krb5kdc[21590](info): set up 2 sockets
Jun 21 05:20:04 dist-test-slave-czm5 krb5kdc[21590](info): commencing operation
krb5kdc: starting...
W20260621 05:20:05.135435 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:05 dist-test-slave-czm5 krb5kdc[21590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:20:05.246376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:05.322359 21603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:05.322546 21603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:05.322566 21603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:05.323685 21603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:05.323722 21603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:05.323734 21603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:05.323772 21603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:05.323800 21603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:05.325191 21603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:05.325424 21603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:05.325635 21603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:05.330314 21608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:05.330456 21603 server_base.cc:1034] running on GCE node
W20260621 05:20:05.330332 21611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:05.330332 21609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:05.330837 21603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:05.331125 21603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:05.332275 21603 hybrid_clock.cc:648] HybridClock initialized: now 1782019205332266 us; error 32 us; skew 500 ppm
Jun 21 05:20:05 dist-test-slave-czm5 krb5kdc[21590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:05.337662 21603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:20:05.338097 21603 webserver.cc:460] Webserver started at http://127.0.0.1:40581/ using document root <none> and password file <none>
I20260621 05:20:05.338318 21603 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:05.338382 21603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:05.338527 21603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:05.339586 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "8225d607b1144cb28c5e586c7576510d"
format_stamp: "Formatted at 2026-06-21 05:20:05 on dist-test-slave-czm5"
server_key: "87290e8d376df4bb7e50417084dd5f2f"
server_key_iv: "116d65dbd6d565bf5b0714a0ffdda5b9"
server_key_version: "encryptionkey@0"
I20260621 05:20:05.340018 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "8225d607b1144cb28c5e586c7576510d"
format_stamp: "Formatted at 2026-06-21 05:20:05 on dist-test-slave-czm5"
server_key: "87290e8d376df4bb7e50417084dd5f2f"
server_key_iv: "116d65dbd6d565bf5b0714a0ffdda5b9"
server_key_version: "encryptionkey@0"
I20260621 05:20:05.341370 21603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:05.342069 21618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:05.342239 21603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:05.342344 21603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "8225d607b1144cb28c5e586c7576510d"
format_stamp: "Formatted at 2026-06-21 05:20:05 on dist-test-slave-czm5"
server_key: "87290e8d376df4bb7e50417084dd5f2f"
server_key_iv: "116d65dbd6d565bf5b0714a0ffdda5b9"
server_key_version: "encryptionkey@0"
I20260621 05:20:05.342434 21603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:05.363476 21603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:05.365352 21603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:05.365509 21603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:05.369695 21603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40273
I20260621 05:20:05.369730 21670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40273 every 8 connection(s)
I20260621 05:20:05.370074 21603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:05.370954 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:05.371033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21603
I20260621 05:20:05.371203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:05.371383 392 external_mini_cluster.cc:1428] Setting key ad0324a71d47de91547a6b5aaef77505
I20260621 05:20:05.374308 21671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d: Bootstrap starting.
I20260621 05:20:05.374971 21671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:05.375252 21671 log.cc:824] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:05.375941 21671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d: No bootstrap required, opened a new log
I20260621 05:20:05.377697 21671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8225d607b1144cb28c5e586c7576510d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260621 05:20:05.377857 21671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 21 05:20:05 dist-test-slave-czm5 krb5kdc[21590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:20:05.377910 21671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8225d607b1144cb28c5e586c7576510d, State: Initialized, Role: FOLLOWER
I20260621 05:20:05.378073 21671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8225d607b1144cb28c5e586c7576510d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260621 05:20:05.378168 21671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:05.378211 21671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:05.378259 21671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:05.378990 21671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8225d607b1144cb28c5e586c7576510d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260621 05:20:05.379105 21671 leader_election.cc:302] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8225d607b1144cb28c5e586c7576510d; no voters:
I20260621 05:20:05.379263 21671 leader_election.cc:288] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:05.379317 21676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:05.379463 21676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [term 1 LEADER]: Becoming Leader. State: Replica: 8225d607b1144cb28c5e586c7576510d, State: Running, Role: LEADER
I20260621 05:20:05.379526 21671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:05.379653 21676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8225d607b1144cb28c5e586c7576510d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260621 05:20:05.380052 21676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8225d607b1144cb28c5e586c7576510d. Latest consensus state: current_term: 1 leader_uuid: "8225d607b1144cb28c5e586c7576510d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8225d607b1144cb28c5e586c7576510d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260621 05:20:05.380066 21678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8225d607b1144cb28c5e586c7576510d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8225d607b1144cb28c5e586c7576510d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260621 05:20:05.380192 21676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:05.380208 21678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:05.380419 21684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:05.380944 21684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:05.382360 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:05.371900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:44713 (local address 127.0.0.1:40273)
0621 05:20:05.372085 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:05.372095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:05.372127 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:20:05.372635 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:05.372643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:05.372708 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:05.372864 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:05.373400 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:05.374732 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:05.375545 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:05.376021 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:05.376156 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:05.378484 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:05.378488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:05.378489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:05.378500 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:05.380622 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:05.381146 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:05.381149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:05.381149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:05.381245 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:05.381469 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:05.381470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:05.381470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:05.381737 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:05.381915 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:05.382003 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:20:05.382115 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260621 05:20:05.382632 21684 catalog_manager.cc:1347] Generated new cluster ID: 9032d0df110c434d90b85dcefefae863
I20260621 05:20:05.382685 21684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:05.383189 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:05.393661 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:05.384605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:59093 (local address 127.0.0.1:40273)
0621 05:20:05.384639 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:05.384641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:05.387792 (+ 3151us) server_negotiation.cc:407] Connection header received
0621 05:20:05.387860 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:05.387861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:05.387870 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:05.387914 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:05.388424 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:05.388893 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:05.389567 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:05.389783 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:05.389867 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:05.391003 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:05.391004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:05.391005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:05.391012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:05.392507 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:05.392864 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:05.392865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:05.392865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:05.392916 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:05.393152 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:05.393152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:05.393153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:05.393260 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:05.393365 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:05.393480 (+ 115us) server_negotiation.cc:299] Negotiation successful
0621 05:20:05.393538 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:20:05.394394 21684 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:05.394869 21684 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:05.405797 21684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8225d607b1144cb28c5e586c7576510d: Generated new TSK 0
I20260621 05:20:05.405992 21684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:05.407917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21603
W20260621 05:20:05.412173 21588 connection.cc:535] client connection to 127.0.0.1:40273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40273): Connection reset by peer (error 104)
2026-06-21T05:20:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:05 dist-test-slave-czm5 krb5kdc[21702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:05 dist-test-slave-czm5 krb5kdc[21702](info): set up 2 sockets
Jun 21 05:20:05 dist-test-slave-czm5 krb5kdc[21702](info): commencing operation
krb5kdc: starting...
W20260621 05:20:06.435596 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:06 dist-test-slave-czm5 krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:20:06.548110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:06.626328 21715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:06.626526 21715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:06.626549 21715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:06.627689 21715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:06.627729 21715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:06.627743 21715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:06.627786 21715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:06.627815 21715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:06.629220 21715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:06.629401 21715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:06.629635 21715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:06.634325 21721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:06.634476 21715 server_base.cc:1034] running on GCE node
W20260621 05:20:06.634346 21720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:06.634536 21723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:06.634831 21715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:06.635133 21715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:06.636268 21715 hybrid_clock.cc:648] HybridClock initialized: now 1782019206636259 us; error 26 us; skew 500 ppm
Jun 21 05:20:06 dist-test-slave-czm5 krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:06.638692 21715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:20:06.639158 21715 webserver.cc:460] Webserver started at http://127.0.0.1:43089/ using document root <none> and password file <none>
I20260621 05:20:06.639412 21715 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:06.639498 21715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:06.639698 21715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:06.640714 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "962c43dd48e449caa368e29ffe769807"
format_stamp: "Formatted at 2026-06-21 05:20:06 on dist-test-slave-czm5"
server_key: "4e2715a0ac7fb348af8985bea12391b1"
server_key_iv: "7301e1296414be4efdcd7a775333ded2"
server_key_version: "encryptionkey@0"
I20260621 05:20:06.641068 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "962c43dd48e449caa368e29ffe769807"
format_stamp: "Formatted at 2026-06-21 05:20:06 on dist-test-slave-czm5"
server_key: "4e2715a0ac7fb348af8985bea12391b1"
server_key_iv: "7301e1296414be4efdcd7a775333ded2"
server_key_version: "encryptionkey@0"
I20260621 05:20:06.642354 21715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:06.643078 21730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:06.643244 21715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:06.643340 21715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "962c43dd48e449caa368e29ffe769807"
format_stamp: "Formatted at 2026-06-21 05:20:06 on dist-test-slave-czm5"
server_key: "4e2715a0ac7fb348af8985bea12391b1"
server_key_iv: "7301e1296414be4efdcd7a775333ded2"
server_key_version: "encryptionkey@0"
I20260621 05:20:06.643420 21715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:06.657835 21715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:06.660203 21715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:06.660388 21715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:06.664322 21715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41909
I20260621 05:20:06.664359 21782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41909 every 8 connection(s)
I20260621 05:20:06.664708 21715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:06.665645 21783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:06.667757 21783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807: Bootstrap starting.
I20260621 05:20:06.668262 21783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:06.668478 21783 log.cc:824] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:06.669081 21783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807: No bootstrap required, opened a new log
I20260621 05:20:06.670527 21783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962c43dd48e449caa368e29ffe769807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260621 05:20:06.670667 21783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:06.670689 21783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962c43dd48e449caa368e29ffe769807, State: Initialized, Role: FOLLOWER
I20260621 05:20:06.670820 21783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962c43dd48e449caa368e29ffe769807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260621 05:20:06.670897 21783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:06.670926 21783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:06.670956 21783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:06.671434 21783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962c43dd48e449caa368e29ffe769807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260621 05:20:06.671514 21783 leader_election.cc:302] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 962c43dd48e449caa368e29ffe769807; no voters:
I20260621 05:20:06.671675 21783 leader_election.cc:288] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:06.671736 21786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:06.671926 21786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [term 1 LEADER]: Becoming Leader. State: Replica: 962c43dd48e449caa368e29ffe769807, State: Running, Role: LEADER
I20260621 05:20:06.671932 21783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:06.672024 21786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962c43dd48e449caa368e29ffe769807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260621 05:20:06.672360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21715
I20260621 05:20:06.672504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:06.672744 392 external_mini_cluster.cc:1428] Setting key 640d3f8a8655996285a3af948b09bb9b
I20260621 05:20:06.673197 21788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 962c43dd48e449caa368e29ffe769807. Latest consensus state: current_term: 1 leader_uuid: "962c43dd48e449caa368e29ffe769807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962c43dd48e449caa368e29ffe769807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } } }
I20260621 05:20:06.673295 21788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:06.673553 21787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "962c43dd48e449caa368e29ffe769807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962c43dd48e449caa368e29ffe769807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } } }
I20260621 05:20:06.673725 21787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:06.673617 21803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:06.674288 21803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:06.675993 21803 catalog_manager.cc:1347] Generated new cluster ID: b16113694a8d4baf8ff10e135a4eec34
I20260621 05:20:06.676085 21803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:06.678771 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:06.673286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:52779 (local address 127.0.0.1:41909)
0621 05:20:06.673463 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:06.673482 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:06.673515 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:20:06.673723 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:06.673730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:06.673787 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:06.673926 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:06.674215 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:06.675495 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:06.676581 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:06.677234 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:06.677353 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:06.678060 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:06.678064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:06.678064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:06.678292 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:06.678330 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:06.678344 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:06.678431 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:06.678523 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 05:20:06.678552 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260621 05:20:06.679746 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:06.685743 21803 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:06.686309 21803 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:06.687250 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:06.681257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:48441 (local address 127.0.0.1:41909)
0621 05:20:06.681290 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:06.681292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:06.683799 (+ 2507us) server_negotiation.cc:407] Connection header received
0621 05:20:06.683829 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:06.683830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:06.683842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:06.683925 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:06.684056 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:06.684981 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:06.685903 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:06.686217 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:06.686285 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:06.686997 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:06.686998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:06.686999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:06.687002 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:06.687016 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:06.687020 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:06.687076 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:06.687165 (+ 89us) server_negotiation.cc:299] Negotiation successful
0621 05:20:06.687176 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":17}
I20260621 05:20:06.691715 21803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 962c43dd48e449caa368e29ffe769807: Generated new TSK 0
I20260621 05:20:06.691887 21803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:06.697229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21715
W20260621 05:20:06.701052 21700 connection.cc:535] client connection to 127.0.0.1:41909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41909): Connection reset by peer (error 104)
2026-06-21T05:20:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:06 dist-test-slave-czm5 krb5kdc[21814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:06 dist-test-slave-czm5 krb5kdc[21814](info): set up 2 sockets
Jun 21 05:20:06 dist-test-slave-czm5 krb5kdc[21814](info): commencing operation
krb5kdc: starting...
W20260621 05:20:07.711990 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:07 dist-test-slave-czm5 krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:20:07.822185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:07.898958 21827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:07.899185 21827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:07.899222 21827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:07.900306 21827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:07.900382 21827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:07.900408 21827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:07.900434 21827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:07.900461 21827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:07.901880 21827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:07.902136 21827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:07.902354 21827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:07.907019 21832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:07.907088 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:07.907089 21833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:07.907187 21827 server_base.cc:1034] running on GCE node
I20260621 05:20:07.907437 21827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:07.907763 21827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:07.908926 21827 hybrid_clock.cc:648] HybridClock initialized: now 1782019207908900 us; error 37 us; skew 500 ppm
Jun 21 05:20:07 dist-test-slave-czm5 krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:07.911218 21827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:20:07.911737 21827 webserver.cc:460] Webserver started at http://127.0.0.1:37785/ using document root <none> and password file <none>
I20260621 05:20:07.911934 21827 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:07.911974 21827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:07.912060 21827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:07.912997 21827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "b0cd34afc1c54cb2abe9fc4306002c84"
format_stamp: "Formatted at 2026-06-21 05:20:07 on dist-test-slave-czm5"
server_key: "365b49689b8dca5feb9860bec5c821ba"
server_key_iv: "6b1dae3a7078fa20f483f4fa66a46f2d"
server_key_version: "encryptionkey@0"
I20260621 05:20:07.913292 21827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0cd34afc1c54cb2abe9fc4306002c84"
format_stamp: "Formatted at 2026-06-21 05:20:07 on dist-test-slave-czm5"
server_key: "365b49689b8dca5feb9860bec5c821ba"
server_key_iv: "6b1dae3a7078fa20f483f4fa66a46f2d"
server_key_version: "encryptionkey@0"
I20260621 05:20:07.914472 21827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:20:07.915129 21842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:07.915277 21827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:20:07.915359 21827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "b0cd34afc1c54cb2abe9fc4306002c84"
format_stamp: "Formatted at 2026-06-21 05:20:07 on dist-test-slave-czm5"
server_key: "365b49689b8dca5feb9860bec5c821ba"
server_key_iv: "6b1dae3a7078fa20f483f4fa66a46f2d"
server_key_version: "encryptionkey@0"
I20260621 05:20:07.915409 21827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:07.918527 21827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:07.920276 21827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:07.920372 21827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:07.923877 21827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45497
I20260621 05:20:07.923930 21894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45497 every 8 connection(s)
I20260621 05:20:07.924216 21827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:07.925112 21895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:07.926522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21827
I20260621 05:20:07.926652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:07.926820 392 external_mini_cluster.cc:1428] Setting key 1c716342b1a7e075c1b24a94efe20b90
I20260621 05:20:07.927578 21895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84: Bootstrap starting.
I20260621 05:20:07.928212 21895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:07.928539 21895 log.cc:824] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:07.929311 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:07.927310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:39997 (local address 127.0.0.1:45497)
0621 05:20:07.927474 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:07.927483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:07.927507 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:20:07.927964 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:07.927971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:07.928005 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:07.928117 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:07.928878 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:07.928883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:07.928883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:07.928888 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:07.928925 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:07.928938 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:07.929034 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:07.929069 (+ 35us) server_negotiation.cc:299] Negotiation successful
0621 05:20:07.929132 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260621 05:20:07.929354 21895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84: No bootstrap required, opened a new log
I20260621 05:20:07.930296 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:07.932019 21895 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0cd34afc1c54cb2abe9fc4306002c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260621 05:20:07.932214 21895 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:07.932253 21895 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0cd34afc1c54cb2abe9fc4306002c84, State: Initialized, Role: FOLLOWER
I20260621 05:20:07.932371 21895 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0cd34afc1c54cb2abe9fc4306002c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260621 05:20:07.932471 21895 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:07.932492 21895 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:07.932595 21895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:07.932945 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:07.931889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.83:37057 (local address 127.0.0.1:45497)
0621 05:20:07.931949 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:07.931951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:07.932107 (+ 156us) server_negotiation.cc:407] Connection header received
0621 05:20:07.932128 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:07.932129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:07.932133 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:07.932186 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:07.932745 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:07.932746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:07.932747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:07.932751 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:07.932764 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:07.932768 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:07.932823 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:07.932857 (+ 34us) server_negotiation.cc:299] Negotiation successful
0621 05:20:07.932868 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:20:07.933343 21895 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0cd34afc1c54cb2abe9fc4306002c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260621 05:20:07.933446 21895 leader_election.cc:302] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0cd34afc1c54cb2abe9fc4306002c84; no voters:
I20260621 05:20:07.933604 21895 leader_election.cc:288] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:07.933660 21905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:07.933806 21905 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [term 1 LEADER]: Becoming Leader. State: Replica: b0cd34afc1c54cb2abe9fc4306002c84, State: Running, Role: LEADER
I20260621 05:20:07.933838 21895 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:07.933952 21905 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0cd34afc1c54cb2abe9fc4306002c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260621 05:20:07.934280 21906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0cd34afc1c54cb2abe9fc4306002c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0cd34afc1c54cb2abe9fc4306002c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } } }
I20260621 05:20:07.934284 21907 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0cd34afc1c54cb2abe9fc4306002c84. Latest consensus state: current_term: 1 leader_uuid: "b0cd34afc1c54cb2abe9fc4306002c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0cd34afc1c54cb2abe9fc4306002c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } } }
I20260621 05:20:07.934382 21906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:07.934389 21907 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84 [sys.catalog]: This master's current role is: LEADER
W20260621 05:20:07.935662 21921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:07.935712 21921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:07.935804 21913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:07.935986 21913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:07.937287 21913 catalog_manager.cc:1347] Generated new cluster ID: 863afb55edbe44a096149da12cacc807
I20260621 05:20:07.937335 21913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:07.946278 21913 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:07.946411 21913 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:07.948889 21913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0cd34afc1c54cb2abe9fc4306002c84: Generated new TSK 0
I20260621 05:20:07.949066 21913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:07.951392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21827
2026-06-21T05:20:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:08 dist-test-slave-czm5 krb5kdc[21926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:08 dist-test-slave-czm5 krb5kdc[21926](info): set up 2 sockets
Jun 21 05:20:08 dist-test-slave-czm5 krb5kdc[21926](info): commencing operation
krb5kdc: starting...
W20260621 05:20:08.971894 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:09 dist-test-slave-czm5 krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:20:09.090461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:09.171731 21939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:09.171960 21939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:09.171985 21939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:09.173136 21939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:09.173180 21939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:09.173194 21939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:09.173206 21939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:09.173218 21939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:09.174665 21939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:09.174929 21939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:09.175154 21939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:09.180276 21944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:09.180269 21945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:09.180302 21939 server_base.cc:1034] running on GCE node
W20260621 05:20:09.180263 21947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:09.180734 21939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:09.181043 21939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:09.182225 21939 hybrid_clock.cc:648] HybridClock initialized: now 1782019209182219 us; error 38 us; skew 500 ppm
Jun 21 05:20:09 dist-test-slave-czm5 krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:09.184804 21939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:20:09.185221 21939 webserver.cc:460] Webserver started at http://127.0.98.62:39073/ using document root <none> and password file <none>
I20260621 05:20:09.185459 21939 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:09.185513 21939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:09.185693 21939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:09.186823 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "2b478eef2e114c27adb9e78d9bd93e06"
format_stamp: "Formatted at 2026-06-21 05:20:09 on dist-test-slave-czm5"
server_key: "2c40f8b673fade77cad3a360a4730d72"
server_key_iv: "da44fd9a4ef766e3328a35e023c7776b"
server_key_version: "encryptionkey@0"
I20260621 05:20:09.187219 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b478eef2e114c27adb9e78d9bd93e06"
format_stamp: "Formatted at 2026-06-21 05:20:09 on dist-test-slave-czm5"
server_key: "2c40f8b673fade77cad3a360a4730d72"
server_key_iv: "da44fd9a4ef766e3328a35e023c7776b"
server_key_version: "encryptionkey@0"
I20260621 05:20:09.188681 21939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:20:09.189435 21954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:09.189630 21939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:09.189741 21939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "2b478eef2e114c27adb9e78d9bd93e06"
format_stamp: "Formatted at 2026-06-21 05:20:09 on dist-test-slave-czm5"
server_key: "2c40f8b673fade77cad3a360a4730d72"
server_key_iv: "da44fd9a4ef766e3328a35e023c7776b"
server_key_version: "encryptionkey@0"
I20260621 05:20:09.189817 21939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:09.198284 21939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:09.200326 21939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:09.200497 21939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:09.204564 21939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39305
I20260621 05:20:09.204627 22006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39305 every 8 connection(s)
I20260621 05:20:09.205039 21939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:09.205489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 21939
I20260621 05:20:09.205641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:09.205854 392 external_mini_cluster.cc:1428] Setting key 066ad29c59d0f45de0f9894a8e592758
I20260621 05:20:09.206189 22007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:09.209833 22007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06: Bootstrap starting.
I20260621 05:20:09.210532 22007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:09.210866 22007 log.cc:824] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:09.211810 22007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06: No bootstrap required, opened a new log
Jun 21 05:20:09 dist-test-slave-czm5 krb5kdc[21926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:20:09.214396 22007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b478eef2e114c27adb9e78d9bd93e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260621 05:20:09.214609 22007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:09.214656 22007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b478eef2e114c27adb9e78d9bd93e06, State: Initialized, Role: FOLLOWER
I20260621 05:20:09.214829 22007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b478eef2e114c27adb9e78d9bd93e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260621 05:20:09.214936 22007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:09.214979 22007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:09.215036 22007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:09.215660 22007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b478eef2e114c27adb9e78d9bd93e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260621 05:20:09.215809 22007 leader_election.cc:302] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b478eef2e114c27adb9e78d9bd93e06; no voters:
I20260621 05:20:09.216040 22007 leader_election.cc:288] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:09.216099 22012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:09.216251 22012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [term 1 LEADER]: Becoming Leader. State: Replica: 2b478eef2e114c27adb9e78d9bd93e06, State: Running, Role: LEADER
I20260621 05:20:09.216323 22007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:09.216348 22012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b478eef2e114c27adb9e78d9bd93e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260621 05:20:09.216790 22012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b478eef2e114c27adb9e78d9bd93e06. Latest consensus state: current_term: 1 leader_uuid: "2b478eef2e114c27adb9e78d9bd93e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b478eef2e114c27adb9e78d9bd93e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260621 05:20:09.217031 22012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:09.216957 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b478eef2e114c27adb9e78d9bd93e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b478eef2e114c27adb9e78d9bd93e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260621 05:20:09.217449 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:09.217593 22021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:09.217840 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:09.206386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59071 (local address 127.0.98.62:39305)
0621 05:20:09.206574 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:09.206585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:09.206619 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:20:09.207172 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:09.207179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:09.207237 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:09.207355 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:09.208240 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:09.209135 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:09.210242 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:09.210692 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:09.210773 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:09.213641 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:09.213645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:09.213645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:09.213657 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:09.215669 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:09.216209 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:09.216214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:09.216214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:09.216326 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:09.216615 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:09.216616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:09.216617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:09.217032 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:09.217215 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:09.217527 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 05:20:09.217644 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260621 05:20:09.218103 22021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:09.218977 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:09.219759 22021 catalog_manager.cc:1347] Generated new cluster ID: 4553dc0dec254c1b8c77b72d358bd144
I20260621 05:20:09.219810 22021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:09.228318 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:09.220629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56647 (local address 127.0.98.62:39305)
0621 05:20:09.220665 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:09.220666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:09.220679 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:20:09.220748 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:09.220749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:09.220758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:09.220810 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:09.221483 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:09.222204 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:09.223238 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:09.223578 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:09.223665 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:09.225285 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:09.225287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:09.225287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:09.225294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:09.226877 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:09.227319 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:09.227320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:09.227320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:09.227374 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:09.227662 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:09.227663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:09.227663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:09.227785 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:09.227873 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:09.228045 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 05:20:09.228117 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:20:09.230685 22021 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:09.231408 22021 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:09.240670 22021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b478eef2e114c27adb9e78d9bd93e06: Generated new TSK 0
I20260621 05:20:09.240906 22021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:09.246220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 21939
W20260621 05:20:09.250639 21924 connection.cc:535] client connection to 127.0.98.62:39305 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39305): Connection reset by peer (error 104)
2026-06-21T05:20:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:09 dist-test-slave-czm5 krb5kdc[22038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:09 dist-test-slave-czm5 krb5kdc[22038](info): set up 2 sockets
Jun 21 05:20:09 dist-test-slave-czm5 krb5kdc[22038](info): commencing operation
krb5kdc: starting...
W20260621 05:20:10.263764 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:10 dist-test-slave-czm5 krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:20:10.379756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:10.462759 22051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:10.463012 22051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:10.463048 22051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:10.464339 22051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:10.464418 22051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:10.464443 22051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:10.464474 22051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:10.464498 22051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:10.465916 22051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:10.466177 22051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:10.466382 22051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:10.470827 22056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:10.471078 22051 server_base.cc:1034] running on GCE node
W20260621 05:20:10.470835 22059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:10.470899 22057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:10.471442 22051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:10.471797 22051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:10.472934 22051 hybrid_clock.cc:648] HybridClock initialized: now 1782019210472926 us; error 27 us; skew 500 ppm
Jun 21 05:20:10 dist-test-slave-czm5 krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:10.478307 22051 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:20:10.478735 22051 webserver.cc:460] Webserver started at http://127.0.98.62:44001/ using document root <none> and password file <none>
I20260621 05:20:10.478967 22051 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:10.479048 22051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:10.479218 22051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:10.480360 22051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "41bd9884bc524b3f9bfb7ef273869b7b"
format_stamp: "Formatted at 2026-06-21 05:20:10 on dist-test-slave-czm5"
server_key: "2b034dd31804229fc0a545fd245b1b2d"
server_key_iv: "f4a7b470ed466485a64846580231cb31"
server_key_version: "encryptionkey@0"
I20260621 05:20:10.480741 22051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "41bd9884bc524b3f9bfb7ef273869b7b"
format_stamp: "Formatted at 2026-06-21 05:20:10 on dist-test-slave-czm5"
server_key: "2b034dd31804229fc0a545fd245b1b2d"
server_key_iv: "f4a7b470ed466485a64846580231cb31"
server_key_version: "encryptionkey@0"
I20260621 05:20:10.482046 22051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:10.482775 22066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:10.482944 22051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:10.483050 22051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "41bd9884bc524b3f9bfb7ef273869b7b"
format_stamp: "Formatted at 2026-06-21 05:20:10 on dist-test-slave-czm5"
server_key: "2b034dd31804229fc0a545fd245b1b2d"
server_key_iv: "f4a7b470ed466485a64846580231cb31"
server_key_version: "encryptionkey@0"
I20260621 05:20:10.483146 22051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:10.499342 22051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:10.501871 22051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:10.502048 22051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:10.505980 22051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33111
I20260621 05:20:10.506053 22118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33111 every 8 connection(s)
I20260621 05:20:10.506421 22051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:10.507308 22119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:10.509569 22119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b: Bootstrap starting.
I20260621 05:20:10.510251 22119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:10.510536 22119 log.cc:824] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:10.511219 22119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b: No bootstrap required, opened a new log
I20260621 05:20:10.512852 22119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bd9884bc524b3f9bfb7ef273869b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33111 } }
I20260621 05:20:10.512991 22119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:10.513053 22119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41bd9884bc524b3f9bfb7ef273869b7b, State: Initialized, Role: FOLLOWER
I20260621 05:20:10.513226 22119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bd9884bc524b3f9bfb7ef273869b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33111 } }
I20260621 05:20:10.513304 22119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:10.513343 22119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:10.513392 22119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:10.513899 22119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bd9884bc524b3f9bfb7ef273869b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33111 } }
I20260621 05:20:10.514008 22119 leader_election.cc:302] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41bd9884bc524b3f9bfb7ef273869b7b; no voters:
I20260621 05:20:10.514200 22119 leader_election.cc:288] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:10.514228 22122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:10.514420 22122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [term 1 LEADER]: Becoming Leader. State: Replica: 41bd9884bc524b3f9bfb7ef273869b7b, State: Running, Role: LEADER
I20260621 05:20:10.514474 22119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:10.514580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22051
I20260621 05:20:10.514565 22122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bd9884bc524b3f9bfb7ef273869b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33111 } }
I20260621 05:20:10.514693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:10.514915 392 external_mini_cluster.cc:1428] Setting key 012967f9322e08b5ea8f6fd70e713107
I20260621 05:20:10.514937 22122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41bd9884bc524b3f9bfb7ef273869b7b. Latest consensus state: current_term: 1 leader_uuid: "41bd9884bc524b3f9bfb7ef273869b7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bd9884bc524b3f9bfb7ef273869b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33111 } } }
I20260621 05:20:10.515002 22122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:10.515240 22123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41bd9884bc524b3f9bfb7ef273869b7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bd9884bc524b3f9bfb7ef273869b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33111 } } }
I20260621 05:20:10.515322 22123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:10.515965 22127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:10.516588 22127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:10.518033 22127 catalog_manager.cc:1347] Generated new cluster ID: b7b9d3877f6a4661a6533cdba20d441c
I20260621 05:20:10.518086 22127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:10.520715 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:10.515365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56979 (local address 127.0.98.62:33111)
0621 05:20:10.515561 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:10.515571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:10.515607 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:20:10.515776 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:10.515784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:10.515848 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:10.516256 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:10.516352 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:10.517755 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:10.518657 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:10.519119 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:10.519261 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:10.520051 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:10.520055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:10.520055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:10.520236 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:10.520265 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:10.520289 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:10.520379 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:10.520474 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 05:20:10.520503 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260621 05:20:10.521595 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:10.525962 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:10.523245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55809 (local address 127.0.98.62:33111)
0621 05:20:10.523298 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:10.523300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:10.523306 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:20:10.523323 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:10.523324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:10.523333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:10.523406 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:10.523535 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:10.524072 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:10.524747 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:10.524982 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:10.525070 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:10.525730 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:10.525731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:10.525731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:10.525735 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:10.525749 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:10.525753 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:10.525804 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:10.525861 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 05:20:10.525872 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:20:10.531610 22127 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:10.532249 22127 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:10.534730 22127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41bd9884bc524b3f9bfb7ef273869b7b: Generated new TSK 0
I20260621 05:20:10.534932 22127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:10.538024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22051
W20260621 05:20:10.541865 22036 connection.cc:535] client connection to 127.0.98.62:33111 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33111): Connection reset by peer (error 104)
2026-06-21T05:20:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:10 dist-test-slave-czm5 krb5kdc[22150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:10 dist-test-slave-czm5 krb5kdc[22150](info): set up 2 sockets
Jun 21 05:20:10 dist-test-slave-czm5 krb5kdc[22150](info): commencing operation
krb5kdc: starting...
W20260621 05:20:11.563731 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:11 dist-test-slave-czm5 krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:20:11.674341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44271
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:11.753535 22163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:11.753737 22163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:11.753759 22163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:11.754987 22163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:11.755031 22163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:11.755046 22163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:11.755059 22163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:11.755102 22163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:11.756636 22163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:11.756856 22163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:11.757098 22163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:11.761902 22171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:11.761909 22168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:11.761909 22169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:11.762142 22163 server_base.cc:1034] running on GCE node
I20260621 05:20:11.762365 22163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:11.762650 22163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:11.763788 22163 hybrid_clock.cc:648] HybridClock initialized: now 1782019211763778 us; error 31 us; skew 500 ppm
Jun 21 05:20:11 dist-test-slave-czm5 krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:11.766171 22163 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:20:11.766597 22163 webserver.cc:460] Webserver started at http://127.0.98.62:39171/ using document root <none> and password file <none>
I20260621 05:20:11.766817 22163 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:11.766882 22163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:11.767022 22163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:11.768107 22163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "6ce8889e9b48498e8cb246856f17098e"
format_stamp: "Formatted at 2026-06-21 05:20:11 on dist-test-slave-czm5"
server_key: "d81ff6f806cc3925b57094309a3fd176"
server_key_iv: "70ab6f975432bc83010e453d7b7865c3"
server_key_version: "encryptionkey@0"
I20260621 05:20:11.768489 22163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ce8889e9b48498e8cb246856f17098e"
format_stamp: "Formatted at 2026-06-21 05:20:11 on dist-test-slave-czm5"
server_key: "d81ff6f806cc3925b57094309a3fd176"
server_key_iv: "70ab6f975432bc83010e453d7b7865c3"
server_key_version: "encryptionkey@0"
I20260621 05:20:11.769791 22163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:11.770473 22178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:11.770650 22163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:11.770746 22163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "6ce8889e9b48498e8cb246856f17098e"
format_stamp: "Formatted at 2026-06-21 05:20:11 on dist-test-slave-czm5"
server_key: "d81ff6f806cc3925b57094309a3fd176"
server_key_iv: "70ab6f975432bc83010e453d7b7865c3"
server_key_version: "encryptionkey@0"
I20260621 05:20:11.770826 22163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:11.774055 22163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:11.776394 22163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:11.776547 22163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:11.780365 22163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44271
I20260621 05:20:11.780392 22230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44271 every 8 connection(s)
I20260621 05:20:11.780727 22163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:11.781706 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:11.784024 22231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e: Bootstrap starting.
I20260621 05:20:11.784584 22231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:11.784857 22231 log.cc:824] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:11.785547 22231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e: No bootstrap required, opened a new log
I20260621 05:20:11.787053 22231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce8889e9b48498e8cb246856f17098e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44271 } }
I20260621 05:20:11.787204 22231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:11.787258 22231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce8889e9b48498e8cb246856f17098e, State: Initialized, Role: FOLLOWER
I20260621 05:20:11.787421 22231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce8889e9b48498e8cb246856f17098e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44271 } }
I20260621 05:20:11.787484 22231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:11.787573 22231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:11.787658 22231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:11.788199 22231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce8889e9b48498e8cb246856f17098e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44271 } }
I20260621 05:20:11.788309 22231 leader_election.cc:302] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ce8889e9b48498e8cb246856f17098e; no voters:
I20260621 05:20:11.788496 22231 leader_election.cc:288] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:11.788551 22234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:11.788719 22234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [term 1 LEADER]: Becoming Leader. State: Replica: 6ce8889e9b48498e8cb246856f17098e, State: Running, Role: LEADER
I20260621 05:20:11.788746 22231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:11.788836 22234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce8889e9b48498e8cb246856f17098e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44271 } }
I20260621 05:20:11.788861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22163
I20260621 05:20:11.788961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:11.789176 392 external_mini_cluster.cc:1428] Setting key f235dcd22ce6130f9f5abe1ab015fb5c
I20260621 05:20:11.789188 22235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ce8889e9b48498e8cb246856f17098e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce8889e9b48498e8cb246856f17098e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44271 } } }
I20260621 05:20:11.789284 22235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:11.789176 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ce8889e9b48498e8cb246856f17098e. Latest consensus state: current_term: 1 leader_uuid: "6ce8889e9b48498e8cb246856f17098e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce8889e9b48498e8cb246856f17098e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44271 } } }
I20260621 05:20:11.789463 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:11.789583 22238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:11.790098 22238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:11.791685 22238 catalog_manager.cc:1347] Generated new cluster ID: e4983b6fb08e4958ba852ca4927569cb
I20260621 05:20:11.791743 22238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:11.792449 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:11.789849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47971 (local address 127.0.98.62:44271)
0621 05:20:11.790409 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:11.790419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:11.790454 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:20:11.790718 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:11.790727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:11.790766 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:11.790922 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:11.792042 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:11.792046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:11.792046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:11.792051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:11.792094 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:11.792111 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:11.792186 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:11.792256 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 05:20:11.792298 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":60,"threads_started":1}
I20260621 05:20:11.793323 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:11.800979 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:11.794626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40415 (local address 127.0.98.62:44271)
0621 05:20:11.794661 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:11.794664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:11.799937 (+ 5273us) server_negotiation.cc:407] Connection header received
0621 05:20:11.800027 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:11.800028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:11.800034 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:11.800089 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:11.800747 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:11.800749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:11.800749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:11.800753 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:11.800769 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:11.800773 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:11.800818 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:11.800877 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 05:20:11.800891 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:20:11.803402 22238 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:11.803560 22238 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:11.809481 22238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ce8889e9b48498e8cb246856f17098e: Generated new TSK 0
I20260621 05:20:11.809684 22238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:11.811853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22163
2026-06-21T05:20:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:11 dist-test-slave-czm5 krb5kdc[22262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:11 dist-test-slave-czm5 krb5kdc[22262](info): set up 2 sockets
Jun 21 05:20:11 dist-test-slave-czm5 krb5kdc[22262](info): commencing operation
krb5kdc: starting...
W20260621 05:20:12.811581 392 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:12 dist-test-slave-czm5 krb5kdc[22262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:20:12.921747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33947
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:12.999047 22275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:12.999287 22275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:12.999322 22275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:13.000404 22275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:13.000478 22275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:13.000500 22275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:13.000522 22275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:13.000551 22275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:13.001989 22275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:13.002276 22275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:13.002503 22275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:13.007332 22281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:13.007329 22280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:13.007329 22283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:13.007453 22275 server_base.cc:1034] running on GCE node
I20260621 05:20:13.007802 22275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:13.008175 22275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:13.009348 22275 hybrid_clock.cc:648] HybridClock initialized: now 1782019213009334 us; error 40 us; skew 500 ppm
Jun 21 05:20:13 dist-test-slave-czm5 krb5kdc[22262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:13.011763 22275 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:20:13.012252 22275 webserver.cc:460] Webserver started at http://127.0.98.62:45879/ using document root <none> and password file <none>
I20260621 05:20:13.012488 22275 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:13.012533 22275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:13.012703 22275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:13.013753 22275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "57643c54a471493cb9fcaf3f333c4dd2"
format_stamp: "Formatted at 2026-06-21 05:20:13 on dist-test-slave-czm5"
server_key: "039562b71f89bf755439d9f4be30763d"
server_key_iv: "380949d7c3a37174d7275c3b580d289d"
server_key_version: "encryptionkey@0"
I20260621 05:20:13.014087 22275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "57643c54a471493cb9fcaf3f333c4dd2"
format_stamp: "Formatted at 2026-06-21 05:20:13 on dist-test-slave-czm5"
server_key: "039562b71f89bf755439d9f4be30763d"
server_key_iv: "380949d7c3a37174d7275c3b580d289d"
server_key_version: "encryptionkey@0"
I20260621 05:20:13.015340 22275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:20:13.016111 22290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:13.016348 22275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:20:13.016444 22275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "57643c54a471493cb9fcaf3f333c4dd2"
format_stamp: "Formatted at 2026-06-21 05:20:13 on dist-test-slave-czm5"
server_key: "039562b71f89bf755439d9f4be30763d"
server_key_iv: "380949d7c3a37174d7275c3b580d289d"
server_key_version: "encryptionkey@0"
I20260621 05:20:13.016511 22275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:13.026759 22275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:13.029096 22275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:13.029271 22275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:13.033165 22275 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33947
I20260621 05:20:13.033231 22342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33947 every 8 connection(s)
I20260621 05:20:13.033564 22275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:13.034358 22343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:13.036322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22275
I20260621 05:20:13.036442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:13.036623 392 external_mini_cluster.cc:1428] Setting key 29bf489d35a3955f7e13f3de941a5c17
I20260621 05:20:13.036661 22343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2: Bootstrap starting.
I20260621 05:20:13.037393 22343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:13.037691 22343 log.cc:824] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:13.038470 22343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2: No bootstrap required, opened a new log
I20260621 05:20:13.040649 22343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57643c54a471493cb9fcaf3f333c4dd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33947 } }
I20260621 05:20:13.040812 22343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:13.040843 22343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57643c54a471493cb9fcaf3f333c4dd2, State: Initialized, Role: FOLLOWER
I20260621 05:20:13.040982 22343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57643c54a471493cb9fcaf3f333c4dd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33947 } }
I20260621 05:20:13.041067 22343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:13.041095 22343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:13.041136 22343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:13.041829 22343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57643c54a471493cb9fcaf3f333c4dd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33947 } }
I20260621 05:20:13.041961 22343 leader_election.cc:302] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57643c54a471493cb9fcaf3f333c4dd2; no voters:
I20260621 05:20:13.042119 22343 leader_election.cc:288] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:13.042191 22348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:13.042346 22348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [term 1 LEADER]: Becoming Leader. State: Replica: 57643c54a471493cb9fcaf3f333c4dd2, State: Running, Role: LEADER
I20260621 05:20:13.042435 22343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:13.042449 22348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57643c54a471493cb9fcaf3f333c4dd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33947 } }
I20260621 05:20:13.042850 22349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57643c54a471493cb9fcaf3f333c4dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57643c54a471493cb9fcaf3f333c4dd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33947 } } }
I20260621 05:20:13.042937 22349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:13.043097 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57643c54a471493cb9fcaf3f333c4dd2. Latest consensus state: current_term: 1 leader_uuid: "57643c54a471493cb9fcaf3f333c4dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57643c54a471493cb9fcaf3f333c4dd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33947 } } }
I20260621 05:20:13.043176 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:13.043562 22355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:13.044076 22355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:20:13 dist-test-slave-czm5 krb5kdc[22262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:20:13.045624 22355 catalog_manager.cc:1347] Generated new cluster ID: 05b95acc9d4f404bbb6bb3a6a3de7da0
I20260621 05:20:13.045681 22355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:13.048826 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:13.037156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60799 (local address 127.0.98.62:33947)
0621 05:20:13.037331 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:13.037338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:13.037363 (+ 25us) server_negotiation.cc:407] Connection header received
0621 05:20:13.037865 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:13.037889 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:13.037954 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:13.038105 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:13.038875 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:13.039836 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:13.040945 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:13.041547 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:13.041662 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:13.045139 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:13.045143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:13.045143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:13.045154 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:13.047147 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:13.047568 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:13.047571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:13.047571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:13.047702 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:13.047917 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:13.047918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:13.047918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:13.048225 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:13.048394 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:13.048472 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:20:13.048619 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260621 05:20:13.049715 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:13.058032 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:13.051113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35451 (local address 127.0.98.62:33947)
0621 05:20:13.051151 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:13.051153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:13.051245 (+ 92us) server_negotiation.cc:407] Connection header received
0621 05:20:13.051296 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:13.051297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:13.051306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:13.051359 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:13.052005 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:13.052732 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:13.053707 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:13.054022 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:13.054076 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:13.055402 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:13.055403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:13.055404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:13.055410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:13.056889 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:13.057290 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:13.057291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:13.057291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:13.057343 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:13.057551 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:13.057551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:13.057551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:13.057661 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:13.057743 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:13.057876 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 05:20:13.057947 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:13.073105 22355 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:13.073650 22355 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:13.081089 22355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57643c54a471493cb9fcaf3f333c4dd2: Generated new TSK 0
I20260621 05:20:13.081311 22355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:13.088722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22275
2026-06-21T05:20:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:13 dist-test-slave-czm5 krb5kdc[22374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:13 dist-test-slave-czm5 krb5kdc[22374](info): set up 2 sockets
Jun 21 05:20:13 dist-test-slave-czm5 krb5kdc[22374](info): commencing operation
krb5kdc: starting...
W20260621 05:20:14.087595 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:14 dist-test-slave-czm5 krb5kdc[22374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:20:14.195616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:14.273630 22387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:14.273823 22387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:14.273845 22387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:14.275019 22387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:14.275059 22387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:14.275074 22387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:14.275087 22387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:14.275152 22387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:14.276576 22387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:14.276827 22387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:14.277081 22387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:14.281798 22392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:14.281777 22393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:14.281776 22395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:14.281860 22387 server_base.cc:1034] running on GCE node
I20260621 05:20:14.282231 22387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:14.282512 22387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:14.283710 22387 hybrid_clock.cc:648] HybridClock initialized: now 1782019214283681 us; error 30 us; skew 500 ppm
Jun 21 05:20:14 dist-test-slave-czm5 krb5kdc[22374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:14.286072 22387 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:20:14.286424 22387 webserver.cc:460] Webserver started at http://127.0.98.62:37429/ using document root <none> and password file <none>
I20260621 05:20:14.286620 22387 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:14.286657 22387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:14.286737 22387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:14.287773 22387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "ca2da49127eb4bd58fa9b62ced448b34"
format_stamp: "Formatted at 2026-06-21 05:20:14 on dist-test-slave-czm5"
server_key: "dd6442ce9c6d1b3e3401a30feb7c3244"
server_key_iv: "c0aa5d6e668acd81c383476f62ae40b0"
server_key_version: "encryptionkey@0"
I20260621 05:20:14.288057 22387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca2da49127eb4bd58fa9b62ced448b34"
format_stamp: "Formatted at 2026-06-21 05:20:14 on dist-test-slave-czm5"
server_key: "dd6442ce9c6d1b3e3401a30feb7c3244"
server_key_iv: "c0aa5d6e668acd81c383476f62ae40b0"
server_key_version: "encryptionkey@0"
I20260621 05:20:14.289217 22387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:14.289877 22402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:14.290089 22387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:14.290158 22387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "ca2da49127eb4bd58fa9b62ced448b34"
format_stamp: "Formatted at 2026-06-21 05:20:14 on dist-test-slave-czm5"
server_key: "dd6442ce9c6d1b3e3401a30feb7c3244"
server_key_iv: "c0aa5d6e668acd81c383476f62ae40b0"
server_key_version: "encryptionkey@0"
I20260621 05:20:14.290251 22387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:14.306020 22387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:14.307761 22387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:14.307910 22387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:14.311601 22387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39565
I20260621 05:20:14.311695 22454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39565 every 8 connection(s)
I20260621 05:20:14.312052 22387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:14.312912 22455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:14.315088 22455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34: Bootstrap starting.
I20260621 05:20:14.315672 22455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:14.315955 22455 log.cc:824] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:14.316588 22455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34: No bootstrap required, opened a new log
I20260621 05:20:14.318158 22455 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2da49127eb4bd58fa9b62ced448b34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } }
I20260621 05:20:14.318311 22455 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:14.318388 22455 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2da49127eb4bd58fa9b62ced448b34, State: Initialized, Role: FOLLOWER
I20260621 05:20:14.318557 22455 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2da49127eb4bd58fa9b62ced448b34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } }
I20260621 05:20:14.318619 22455 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:14.318660 22455 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:14.318710 22455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:14.319231 22455 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2da49127eb4bd58fa9b62ced448b34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } }
I20260621 05:20:14.319355 22455 leader_election.cc:302] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2da49127eb4bd58fa9b62ced448b34; no voters:
I20260621 05:20:14.319514 22455 leader_election.cc:288] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:14.319545 22458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:14.319726 22458 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [term 1 LEADER]: Becoming Leader. State: Replica: ca2da49127eb4bd58fa9b62ced448b34, State: Running, Role: LEADER
I20260621 05:20:14.319780 22455 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:14.319828 22458 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2da49127eb4bd58fa9b62ced448b34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } }
I20260621 05:20:14.320174 22460 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2da49127eb4bd58fa9b62ced448b34. Latest consensus state: current_term: 1 leader_uuid: "ca2da49127eb4bd58fa9b62ced448b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2da49127eb4bd58fa9b62ced448b34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } } }
I20260621 05:20:14.320274 22460 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:14.320171 22459 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2da49127eb4bd58fa9b62ced448b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2da49127eb4bd58fa9b62ced448b34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } } }
I20260621 05:20:14.320398 22459 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:14.320421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22387
I20260621 05:20:14.320477 22462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:14.320544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:14.320765 392 external_mini_cluster.cc:1428] Setting key f74e68e4b64731141e2b8925c156186e
I20260621 05:20:14.320933 22462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:14.324555 22462 catalog_manager.cc:1347] Generated new cluster ID: aa843439b58a49a2aee90f4c2e6c227e
I20260621 05:20:14.324606 22462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:14.326283 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:14.321221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55847 (local address 127.0.98.62:39565)
0621 05:20:14.321710 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:14.321718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:14.321785 (+ 67us) server_negotiation.cc:407] Connection header received
0621 05:20:14.321895 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:14.321904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:14.321946 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:14.322057 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:14.322316 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:14.323208 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:14.324358 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:14.324943 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:14.325045 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:14.325685 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:14.325688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:14.325689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:14.325865 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:14.325898 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:14.325910 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:14.325964 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:14.326056 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 05:20:14.326108 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":60,"threads_started":1}
I20260621 05:20:14.327154 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:14.331785 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:14.328614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52173 (local address 127.0.98.62:39565)
0621 05:20:14.328643 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:14.328645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:14.328840 (+ 195us) server_negotiation.cc:407] Connection header received
0621 05:20:14.328865 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:14.328865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:14.328875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:14.328936 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:14.329071 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:14.329763 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:14.330638 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:14.330916 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:14.330984 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:14.331550 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:14.331551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:14.331552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:14.331555 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:14.331569 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:14.331573 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:14.331652 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:14.331704 (+ 52us) server_negotiation.cc:299] Negotiation successful
0621 05:20:14.331725 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:20:14.333418 22462 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:14.333891 22462 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:14.335983 22462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2da49127eb4bd58fa9b62ced448b34: Generated new TSK 0
I20260621 05:20:14.336148 22462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:14.339888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22387
2026-06-21T05:20:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:14 dist-test-slave-czm5 krb5kdc[22486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:14 dist-test-slave-czm5 krb5kdc[22486](info): set up 2 sockets
Jun 21 05:20:14 dist-test-slave-czm5 krb5kdc[22486](info): commencing operation
krb5kdc: starting...
W20260621 05:20:15.355861 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:15 dist-test-slave-czm5 krb5kdc[22486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:20:15.470432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44277
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:15.550243 22499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:15.550477 22499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:15.550514 22499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:15.551642 22499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:15.551726 22499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:15.551751 22499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:15.551781 22499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:15.551812 22499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:15.553315 22499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:15.553588 22499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:15.553802 22499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:15.558547 22507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:15.558559 22504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:15.558552 22505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:15.558881 22499 server_base.cc:1034] running on GCE node
I20260621 05:20:15.559079 22499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:15.559325 22499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:15.560488 22499 hybrid_clock.cc:648] HybridClock initialized: now 1782019215560475 us; error 39 us; skew 500 ppm
Jun 21 05:20:15 dist-test-slave-czm5 krb5kdc[22486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:15.568890 22499 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:20:15.569346 22499 webserver.cc:460] Webserver started at http://127.0.98.62:39637/ using document root <none> and password file <none>
I20260621 05:20:15.569589 22499 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:15.569659 22499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:15.569844 22499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:15.570919 22499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d"
format_stamp: "Formatted at 2026-06-21 05:20:15 on dist-test-slave-czm5"
server_key: "b32108e35733795d8549b6e002257e43"
server_key_iv: "08b0bd16d7a66ae1524466b53989cd49"
server_key_version: "encryptionkey@0"
I20260621 05:20:15.571286 22499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d"
format_stamp: "Formatted at 2026-06-21 05:20:15 on dist-test-slave-czm5"
server_key: "b32108e35733795d8549b6e002257e43"
server_key_iv: "08b0bd16d7a66ae1524466b53989cd49"
server_key_version: "encryptionkey@0"
I20260621 05:20:15.572705 22499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:20:15.573431 22514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:15.573592 22499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:15.573693 22499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d"
format_stamp: "Formatted at 2026-06-21 05:20:15 on dist-test-slave-czm5"
server_key: "b32108e35733795d8549b6e002257e43"
server_key_iv: "08b0bd16d7a66ae1524466b53989cd49"
server_key_version: "encryptionkey@0"
I20260621 05:20:15.573772 22499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:15.577332 22499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:15.579537 22499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:15.579718 22499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:15.583431 22499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44277
I20260621 05:20:15.583482 22566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44277 every 8 connection(s)
I20260621 05:20:15.583832 22499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:15.584836 22567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:15.585271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22499
I20260621 05:20:15.585395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:15.585575 392 external_mini_cluster.cc:1428] Setting key 990b22c97d195377af639cca280f5469
I20260621 05:20:15.587803 22567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d: Bootstrap starting.
I20260621 05:20:15.588018 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:15.586088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48283 (local address 127.0.98.62:44277)
0621 05:20:15.586270 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:15.586279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:15.586312 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:20:15.586805 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:15.586813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:15.586846 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:15.587002 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:15.587553 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:15.587557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:15.587557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:15.587562 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:15.587603 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:15.587637 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:15.587707 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:15.587758 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 05:20:15.587795 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260621 05:20:15.588434 22567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:15.588788 22567 log.cc:824] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:15.588941 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:15.589545 22567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d: No bootstrap required, opened a new log
I20260621 05:20:15.591449 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:15.590535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33083 (local address 127.0.98.62:44277)
0621 05:20:15.590573 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:15.590575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:15.590678 (+ 103us) server_negotiation.cc:407] Connection header received
0621 05:20:15.590700 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:15.590701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:15.590706 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:15.590776 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:15.591258 (+ 482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:15.591259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:15.591259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:15.591264 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:15.591279 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:15.591283 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:15.591347 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:15.591379 (+ 32us) server_negotiation.cc:299] Negotiation successful
0621 05:20:15.591391 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:15.592162 22567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44277 } }
I20260621 05:20:15.592285 22567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:15.592307 22567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f2a0a5ebf42bea5fe1a8ff6d1577d, State: Initialized, Role: FOLLOWER
I20260621 05:20:15.592442 22567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44277 } }
I20260621 05:20:15.592528 22567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:15.592573 22567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:15.592621 22567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:15.593111 22567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44277 } }
I20260621 05:20:15.593226 22567 leader_election.cc:302] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 522f2a0a5ebf42bea5fe1a8ff6d1577d; no voters:
I20260621 05:20:15.593430 22567 leader_election.cc:288] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:15.593499 22577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:15.593649 22567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:15.593652 22577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [term 1 LEADER]: Becoming Leader. State: Replica: 522f2a0a5ebf42bea5fe1a8ff6d1577d, State: Running, Role: LEADER
I20260621 05:20:15.593782 22577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44277 } }
I20260621 05:20:15.594135 22578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44277 } } }
I20260621 05:20:15.594379 22578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:15.594306 22579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 522f2a0a5ebf42bea5fe1a8ff6d1577d. Latest consensus state: current_term: 1 leader_uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522f2a0a5ebf42bea5fe1a8ff6d1577d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44277 } } }
I20260621 05:20:15.594543 22579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:15.594775 22586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:15.595258 22586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:15.596760 22586 catalog_manager.cc:1347] Generated new cluster ID: 710bec51b97249bda707847ce31e1ff6
I20260621 05:20:15.596822 22586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:15.617671 22586 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:15.617849 22586 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:15.623050 22586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 522f2a0a5ebf42bea5fe1a8ff6d1577d: Generated new TSK 0
I20260621 05:20:15.623234 22586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:15.628403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22499
2026-06-21T05:20:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1289 ms)
[----------] 18 tests from AuthTokenIssuingTest (23075 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:15 dist-test-slave-czm5 krb5kdc[22598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:15 dist-test-slave-czm5 krb5kdc[22598](info): set up 2 sockets
Jun 21 05:20:15 dist-test-slave-czm5 krb5kdc[22598](info): commencing operation
krb5kdc: starting...
W20260621 05:20:16.635782 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:16 dist-test-slave-czm5 krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:20:16.748319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33459,127.0.98.61:40257,127.0.98.60:38175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:16.827155 22611 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:20:16.827353 22611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:16.827369 22611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:16.827406 22611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:16.828544 22611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:16.828593 22611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:16.828606 22611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:16.828617 22611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:16.828629 22611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:16.830030 22611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33459,127.0.98.61:40257,127.0.98.60:38175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:16.830235 22611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:16.830405 22611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:16.835065 22619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:16.835031 22617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:16.835044 22616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:16.835207 22611 server_base.cc:1034] running on GCE node
I20260621 05:20:16.835466 22611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:16.835778 22611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:16.836926 22611 hybrid_clock.cc:648] HybridClock initialized: now 1782019216836917 us; error 27 us; skew 500 ppm
Jun 21 05:20:16 dist-test-slave-czm5 krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:16.839300 22611 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:20:16.839835 22611 webserver.cc:460] Webserver started at http://127.0.98.62:32855/ using document root <none> and password file <none>
I20260621 05:20:16.840065 22611 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:16.840135 22611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:16.840296 22611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:16.841320 22611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce"
format_stamp: "Formatted at 2026-06-21 05:20:16 on dist-test-slave-czm5"
server_key: "cf6ae67ddf9b9b8b2c3eedfdf9daea47"
server_key_iv: "c91be64189261e0fc171f073e61763d0"
server_key_version: "encryptionkey@0"
I20260621 05:20:16.841678 22611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce"
format_stamp: "Formatted at 2026-06-21 05:20:16 on dist-test-slave-czm5"
server_key: "cf6ae67ddf9b9b8b2c3eedfdf9daea47"
server_key_iv: "c91be64189261e0fc171f073e61763d0"
server_key_version: "encryptionkey@0"
I20260621 05:20:16.842980 22611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:20:16.843779 22626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:16.843984 22611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:20:16.844082 22611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce"
format_stamp: "Formatted at 2026-06-21 05:20:16 on dist-test-slave-czm5"
server_key: "cf6ae67ddf9b9b8b2c3eedfdf9daea47"
server_key_iv: "c91be64189261e0fc171f073e61763d0"
server_key_version: "encryptionkey@0"
I20260621 05:20:16.844172 22611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:16.860059 22611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:16.861871 22611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:16.862038 22611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:16.865972 22611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33459
I20260621 05:20:16.866004 22678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33459 every 8 connection(s)
I20260621 05:20:16.866343 22611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:16.867314 22679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:16.868815 22679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:16.873100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22611
I20260621 05:20:16.873215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:16.873404 392 external_mini_cluster.cc:1428] Setting key e540cc57f5b1b1a10614c7d7d3f0c06d
Jun 21 05:20:16 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:20:16.878698 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:16.869447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33459 (local address 127.0.0.1:55666)
0621 05:20:16.869592 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:16.869638 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:16.870069 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:16.870341 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:16.870343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:16.870590 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:16.870740 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:16.870744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:16.871725 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:16.871726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:16.872412 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:16.872414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:16.872866 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:16.872867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:16.872919 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:16.873508 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:16.873518 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:16.875069 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:16.877259 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:16.877266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:16.877267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:16.877460 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:16.877726 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:16.877727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:16.877727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:16.877804 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:16.878276 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:16.878280 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:16.878334 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 05:20:16.878406 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:20:16.878534 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260621 05:20:16.878813 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:16.869502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.0.98.62:33459)
0621 05:20:16.869670 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:16.869674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:16.869701 (+ 27us) server_negotiation.cc:407] Connection header received
0621 05:20:16.870217 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:16.870219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:16.870235 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:16.870320 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:16.870817 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:16.871643 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:16.872483 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:16.872779 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:16.872854 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:16.875199 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:16.875205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:16.875205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:16.875218 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:16.877164 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:16.877541 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:16.877544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:16.877545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:16.877655 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:16.877873 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:16.877874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:16.877875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:16.878201 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:16.878299 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:16.878466 (+ 167us) server_negotiation.cc:299] Negotiation successful
0621 05:20:16.878625 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260621 05:20:16.879575 22679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:16.879966 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:16.879850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:55972)
0621 05:20:16.879899 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:16.879941 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260621 05:20:16.880133 22628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
I20260621 05:20:16.880565 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:16.880474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:55984)
0621 05:20:16.880526 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:16.880549 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:20:16.880699 22679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260621 05:20:16.897784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40257
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33459,127.0.98.61:40257,127.0.98.60:38175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:20:16.930032 22679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } attempt: 1
I20260621 05:20:16.930586 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:16.930425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:55990)
0621 05:20:16.930517 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:16.930558 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 05:20:16.930955 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:16.930868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:55994)
0621 05:20:16.930909 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:16.930933 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 05:20:16.931041 22679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
W20260621 05:20:16.980085 22686 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:20:16.980295 22686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:16.980312 22686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:16.980332 22686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:16.981493 22686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:16.981536 22686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:16.981550 22686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:16.981606 22686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:16.981633 22686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:16.983179 22686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33459,127.0.98.61:40257,127.0.98.60:38175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:16.983436 22686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:16.983700 22686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:16.988497 22691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:16.988487 22694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:16.988487 22692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:16.988884 22686 server_base.cc:1034] running on GCE node
I20260621 05:20:16.989176 22686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:16.989464 22686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:16.990631 22686 hybrid_clock.cc:648] HybridClock initialized: now 1782019216990616 us; error 48 us; skew 500 ppm
Jun 21 05:20:16 dist-test-slave-czm5 krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:16.993221 22686 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260621 05:20:16.993657 22686 webserver.cc:460] Webserver started at http://127.0.98.61:42183/ using document root <none> and password file <none>
I20260621 05:20:16.993909 22686 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:16.993957 22686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:16.994100 22686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:16.995147 22686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/data/instance:
uuid: "1335d8cc12314ac8aaa14ff439d63fed"
format_stamp: "Formatted at 2026-06-21 05:20:16 on dist-test-slave-czm5"
server_key: "6ba80e0400b5e40c5abdc5236107a705"
server_key_iv: "b78b83c3aa1b4be2f94035e2a781c456"
server_key_version: "encryptionkey@0"
I20260621 05:20:16.995491 22686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/wal/instance:
uuid: "1335d8cc12314ac8aaa14ff439d63fed"
format_stamp: "Formatted at 2026-06-21 05:20:16 on dist-test-slave-czm5"
server_key: "6ba80e0400b5e40c5abdc5236107a705"
server_key_iv: "b78b83c3aa1b4be2f94035e2a781c456"
server_key_version: "encryptionkey@0"
I20260621 05:20:16.996865 22686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:16.997612 22701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:16.997800 22686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:20:16.997907 22686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/wal
uuid: "1335d8cc12314ac8aaa14ff439d63fed"
format_stamp: "Formatted at 2026-06-21 05:20:16 on dist-test-slave-czm5"
server_key: "6ba80e0400b5e40c5abdc5236107a705"
server_key_iv: "b78b83c3aa1b4be2f94035e2a781c456"
server_key_version: "encryptionkey@0"
I20260621 05:20:16.997980 22686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:16.999198 22679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } attempt: 2
I20260621 05:20:16.999577 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:16.999448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:55998)
0621 05:20:16.999509 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:16.999547 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 05:20:16.999971 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:16.999891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:56008)
0621 05:20:16.999932 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:16.999956 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:20:17.000088 22679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40257: connect: Connection refused (error 111)
I20260621 05:20:17.016762 22686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:17.019147 22686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:17.019268 22686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:17.023219 22686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40257
I20260621 05:20:17.023254 22753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40257 every 8 connection(s)
I20260621 05:20:17.023610 22686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
I20260621 05:20:17.024684 22754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:17.026433 22754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:20:17.032845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22686
I20260621 05:20:17.032943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-1/wal/instance
I20260621 05:20:17.033142 392 external_mini_cluster.cc:1428] Setting key 4182242e2a9fce267097ef094b2d8d2f
I20260621 05:20:17.035600 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.027072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.98.62:33459)
0621 05:20:17.027167 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.027170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.027352 (+ 182us) server_negotiation.cc:407] Connection header received
0621 05:20:17.027913 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.027914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.027927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.027971 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.028623 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.029107 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.030079 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.030288 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.030373 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.032669 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.032671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.032671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.032681 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.034186 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.034688 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.034689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.034689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.034753 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.034984 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.034985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.034986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.035127 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.035195 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.035425 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.035511 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 05:20:17.035812 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.027041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33459 (local address 127.0.0.1:55670)
0621 05:20:17.027233 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.027267 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:17.027805 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:17.028047 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:17.028050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:17.028316 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:17.028510 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.028514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.029209 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.029212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.029989 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.029990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.030394 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.030395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.030514 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.030972 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:17.030979 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:17.032558 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:17.034280 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.034287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.034289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.034606 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.034822 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.034822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.034822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.034917 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.035190 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:17.035194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:17.035302 (+ 108us) client_negotiation.cc:769] Sending connection context
0621 05:20:17.035382 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:20:17.035516 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260621 05:20:17.036355 22754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260621 05:20:17.043965 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.036609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:56010)
0621 05:20:17.036654 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.036664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:17.036700 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:17.036988 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:17.036989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:17.037148 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:17.037211 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.037213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.038018 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.038019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.038949 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.038952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.039409 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.039410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.039487 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.039895 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:17.039902 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:17.041170 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:17.042971 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.042972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.042973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.043126 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.043312 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.043313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.043313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.043361 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.043724 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:17.043725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:17.043760 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 05:20:17.043815 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 05:20:17.043897 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:20:17.044107 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.036703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.0.98.61:40257)
0621 05:20:17.036851 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.036855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.036869 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:20:17.036884 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.036885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.036895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.036962 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.037289 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.037934 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.039035 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.039326 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.039378 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.041259 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.041269 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.041269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.041277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.042899 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.043200 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.043203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.043203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.043254 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.043413 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.043414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.043414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.043614 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.043752 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.043840 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.043955 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260621 05:20:17.044739 22754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:17.045076 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.044967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56410)
0621 05:20:17.045014 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.045058 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:20:17.045215 22703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
I20260621 05:20:17.045640 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.045550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56418)
0621 05:20:17.045594 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.045618 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:20:17.045835 22754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260621 05:20:17.058542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38175
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33459,127.0.98.61:40257,127.0.98.60:38175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:20:17.091274 22679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } attempt: 3
I20260621 05:20:17.095081 22754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attempt: 1
I20260621 05:20:17.095472 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.095352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56422)
0621 05:20:17.095410 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.095445 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:20:17.095868 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.095771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56432)
0621 05:20:17.095817 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.095841 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260621 05:20:17.095963 22754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260621 05:20:17.098958 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.091799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.0.98.61:40257)
0621 05:20:17.091869 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.091871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.091883 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:20:17.091961 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.091961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.091971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.092033 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.092414 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.092909 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.093967 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.094169 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.094221 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.096399 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.096400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.096400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.096408 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.097849 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.098205 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.098207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.098207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.098265 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.098460 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.098461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.098461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.098575 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.098650 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.098796 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.098871 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:20:17.098964 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.091757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:56016)
0621 05:20:17.091813 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.091826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:17.091893 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:17.092059 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:17.092059 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:17.092263 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:17.092340 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.092342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.093009 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.093010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.093869 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.093873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.094252 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.094253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.094348 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.094815 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:17.094821 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:17.096288 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:17.097951 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.097952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.097952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.098120 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.098335 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.098336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.098336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.098399 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.098642 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:17.098643 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:17.098687 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:20:17.098774 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 05:20:17.098872 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260621 05:20:17.099320 22679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:17.099663 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.099548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56442)
0621 05:20:17.099595 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.099640 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:20:17.099962 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.099865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56454)
0621 05:20:17.099909 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.099944 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:20:17.100059 22679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
I20260621 05:20:17.131217 22679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attempt: 1
I20260621 05:20:17.131705 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.131528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56470)
0621 05:20:17.131595 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.131678 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 05:20:17.132078 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.131945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56478)
0621 05:20:17.132004 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.132047 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 05:20:17.132194 22679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
W20260621 05:20:17.140477 22761 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:20:17.140657 22761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:17.140673 22761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:17.140690 22761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:17.141727 22761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:17.141759 22761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:17.141773 22761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:17.141785 22761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:17.141798 22761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:17.143214 22761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33459,127.0.98.61:40257,127.0.98.60:38175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:17.143407 22761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:17.143572 22761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:17.148589 22766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:17.148617 22767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:17.148629 22769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:17.148780 22761 server_base.cc:1034] running on GCE node
I20260621 05:20:17.148998 22761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:17.149291 22761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:17.150431 22761 hybrid_clock.cc:648] HybridClock initialized: now 1782019217150419 us; error 28 us; skew 500 ppm
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:17.152854 22761 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260621 05:20:17.153319 22761 webserver.cc:460] Webserver started at http://127.0.98.60:39679/ using document root <none> and password file <none>
I20260621 05:20:17.153515 22761 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:17.153555 22761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:17.153644 22761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:17.154639 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/data/instance:
uuid: "e5cbf8c032bd4806839ff80fc8a41c5d"
format_stamp: "Formatted at 2026-06-21 05:20:17 on dist-test-slave-czm5"
server_key: "43627af64c8f79ece8b665c3c17aee2c"
server_key_iv: "a902a5e965d79784f71537b8a4a64b05"
server_key_version: "encryptionkey@0"
I20260621 05:20:17.154934 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/wal/instance:
uuid: "e5cbf8c032bd4806839ff80fc8a41c5d"
format_stamp: "Formatted at 2026-06-21 05:20:17 on dist-test-slave-czm5"
server_key: "43627af64c8f79ece8b665c3c17aee2c"
server_key_iv: "a902a5e965d79784f71537b8a4a64b05"
server_key_version: "encryptionkey@0"
I20260621 05:20:17.156270 22761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:17.156989 22776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:17.157207 22761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:17.157282 22761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/wal
uuid: "e5cbf8c032bd4806839ff80fc8a41c5d"
format_stamp: "Formatted at 2026-06-21 05:20:17 on dist-test-slave-czm5"
server_key: "43627af64c8f79ece8b665c3c17aee2c"
server_key_iv: "a902a5e965d79784f71537b8a4a64b05"
server_key_version: "encryptionkey@0"
I20260621 05:20:17.157366 22761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:17.164103 22754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attempt: 2
I20260621 05:20:17.164485 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.164360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56494)
0621 05:20:17.164419 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.164456 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 05:20:17.164685 22761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:17.164844 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.164746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56504)
0621 05:20:17.164791 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.164815 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:20:17.164928 22754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
I20260621 05:20:17.166739 22761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:17.166887 22761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:17.170818 22761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38175
I20260621 05:20:17.170874 22828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38175 every 8 connection(s)
I20260621 05:20:17.171231 22761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
I20260621 05:20:17.172305 22829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:17.173027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22761
I20260621 05:20:17.173110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018933478224-392-0/minicluster-data/master-2/wal/instance
I20260621 05:20:17.173308 392 external_mini_cluster.cc:1428] Setting key 694850dc66a553c6c29c4fe9eb50c406
I20260621 05:20:17.174355 22829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:20:17.184293 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.175252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.0.98.62:33459)
0621 05:20:17.175297 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.175299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.175457 (+ 158us) server_negotiation.cc:407] Connection header received
0621 05:20:17.175530 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.175531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.175542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.175604 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.176105 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.176791 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.178057 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.178329 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.178404 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.180725 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.180727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.180727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.180736 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.182817 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.183217 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.183219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.183219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.183278 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.183541 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.183542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.183542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.183726 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.183782 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.184094 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.184178 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:20:17.184567 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.175192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33459 (local address 127.0.0.1:55672)
0621 05:20:17.175363 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.175393 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:17.175476 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:17.175655 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:17.175658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:17.175919 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:17.176012 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.176017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.176892 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.176893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.177984 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.177987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.178443 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.178444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.178508 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.179015 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:17.179026 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:17.180636 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:17.182902 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.182907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.182909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.183143 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.183343 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.183344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.183344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.183467 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.183793 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:17.183798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:17.183958 (+ 160us) client_negotiation.cc:769] Sending connection context
0621 05:20:17.184052 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 05:20:17.184217 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260621 05:20:17.184567 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.173830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.0.98.60:38175)
0621 05:20:17.174023 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.174033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.174066 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:20:17.174603 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.174612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.174676 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.174833 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.175583 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.176809 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.177910 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.178332 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.178444 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.181291 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.181297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.181298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.181304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.182953 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.183334 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.183338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.183338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.183467 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.183698 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.183699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.183699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.183964 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.184100 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.184206 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.184262 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260621 05:20:17.185424 22829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:20:17.192397 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.185662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40257 (local address 127.0.0.1:56028)
0621 05:20:17.185701 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.185710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:17.185740 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:17.185892 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:17.185893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:17.185998 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:17.186036 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.186037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.186740 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.186741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.187299 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.187301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.187783 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.187784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.187854 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.188380 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:17.188385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:17.189518 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:17.191356 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.191357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.191357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.191510 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.191740 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.191740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.191741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.191790 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.192070 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:17.192083 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:17.192128 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 05:20:17.192200 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:20:17.192276 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260621 05:20:17.192411 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.185726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.0.98.61:40257)
0621 05:20:17.185765 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.185766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.185775 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:17.185814 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.185815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.185824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.185891 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.186106 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.186664 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.187377 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.187711 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.187771 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.189603 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.189605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.189605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.189613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.191284 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.191582 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.191583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.191583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.191658 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.191867 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.191868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.191868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.192001 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.192052 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.192239 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.192324 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:20:17.193020 22829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:17.193145 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.185545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.0.98.62:33459)
0621 05:20:17.185595 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.185596 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.185605 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:17.185625 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.185626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.185634 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.185681 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.186335 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.186825 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.187523 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.187759 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.187827 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.190175 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.190176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.190177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.190184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.191850 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.192198 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.192199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.192200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.192255 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.192458 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.192459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.192460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.192585 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.192898 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.192974 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.193054 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260621 05:20:17.199734 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.193281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56530)
0621 05:20:17.193310 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.193319 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:17.193359 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:17.193486 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:17.193486 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:17.193609 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:17.193631 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.193632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.194401 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.194402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.194881 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.194882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.195283 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.195284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.195330 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.195731 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:17.195736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:17.196981 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:17.198677 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.198678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.198678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.198898 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.199084 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.199085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.199085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.199160 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.199436 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:17.199437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:17.199470 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 05:20:17.199523 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:20:17.199585 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260621 05:20:17.199803 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.193312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.0.98.60:38175)
0621 05:20:17.193344 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.193346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.193353 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:20:17.193424 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.193425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.193433 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.193482 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.193746 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.194326 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.194986 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.195202 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.195284 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.197071 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.197073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.197073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.197080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.198597 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.198964 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.198965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.198965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.199019 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.199219 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.199219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.199220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.199357 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.199427 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.199571 (+ 144us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.199677 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:20:17.201115 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.193746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.0.98.61:40257)
0621 05:20:17.193784 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.193786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.193820 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:20:17.193883 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.193884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.193892 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.193941 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.194442 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.194874 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.195563 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.195889 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.195972 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.198346 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.198348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.198349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.198356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.200040 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.200366 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.200367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.200368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.200419 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.200613 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.200614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.200615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.200729 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.200916 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.200952 (+ 36us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.201023 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:20:17.201337 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d: Bootstrap starting.
I20260621 05:20:17.202415 22829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:17.202625 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:17.202751 22829 log.cc:824] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:17.203615 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d: No bootstrap required, opened a new log
I20260621 05:20:17.203583 22653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55678
I20260621 05:20:17.205686 22829 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260621 05:20:17.205842 22829 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cbf8c032bd4806839ff80fc8a41c5d, State: Initialized, Role: FOLLOWER
I20260621 05:20:17.205920 22829 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260621 05:20:17.206214 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.206259 22829 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:17.206331 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:17.207337 22679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attempt: 2
W20260621 05:20:17.208123 22846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:17.208181 22846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:17.208635 22761 master_runner.cc:186] Error getting master registration for 127.0.98.62:33459: OK, instance_id { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" instance_seqno: 1782019216864869 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:20:17.210992 22761 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:20:17.215171 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.207662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56542)
0621 05:20:17.207713 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.207723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:17.207764 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:17.207997 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:17.207998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:17.208174 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:17.208242 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.208245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.209086 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.209087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.209877 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.209878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.210233 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.210234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.210292 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.210718 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:17.210724 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:17.212161 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:17.214030 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.214030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.214031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.214279 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.214491 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.214492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.214492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.214553 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.214861 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:17.214862 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:17.214909 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 05:20:17.214974 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 05:20:17.215071 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 05:20:17.215179 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.207839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.0.98.60:38175)
0621 05:20:17.207874 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.207876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.207883 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:20:17.207903 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.207903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.207912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.207956 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.208314 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.208982 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.209946 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.210144 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.210234 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.212274 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.212276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.212276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.212284 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.213955 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.214357 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.214358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.214358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.214420 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.214622 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.214623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.214623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.214794 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.214865 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.215016 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.215093 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:20:17.216883 22679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce: Bootstrap starting.
I20260621 05:20:17.217684 22679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:17.218009 22679 log.cc:824] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:17.218890 22679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce: No bootstrap required, opened a new log
I20260621 05:20:17.220407 22679 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260621 05:20:17.220552 22679 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0b1c3e2c5463c870dbb19ef72d6ce, State: Initialized, Role: FOLLOWER
I20260621 05:20:17.220638 22679 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260621 05:20:17.220964 22679 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:17.220947 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.221024 22653 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 05:20:17.221055 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:17.221086 22653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:17.221807 22653 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260621 05:20:17.221997 22653 leader_election.cc:288] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [CANDIDATE]: Term 1 election: Requested vote from peers 1335d8cc12314ac8aaa14ff439d63fed (127.0.98.61:40257), e5cbf8c032bd4806839ff80fc8a41c5d (127.0.98.60:38175)
I20260621 05:20:17.222582 22803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d"
I20260621 05:20:17.222719 22803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:17.223565 22803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0b1c3e2c5463c870dbb19ef72d6ce in term 1.
W20260621 05:20:17.224867 22860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:17.224968 22860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:17.223724 22611 master_runner.cc:186] Error getting master registration for 127.0.98.61:40257: OK, instance_id { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" instance_seqno: 1782019217022064 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:20:17.226660 22611 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:20:17.226840 22628 leader_election.cc:302] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c0b1c3e2c5463c870dbb19ef72d6ce, e5cbf8c032bd4806839ff80fc8a41c5d; no voters:
I20260621 05:20:17.226964 22849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:17.227113 22849 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [term 1 LEADER]: Becoming Leader. State: Replica: e2c0b1c3e2c5463c870dbb19ef72d6ce, State: Running, Role: LEADER
I20260621 05:20:17.227205 22849 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260621 05:20:17.227644 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c0b1c3e2c5463c870dbb19ef72d6ce. Latest consensus state: current_term: 1 leader_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.227733 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:17.232767 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.223369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.0.98.61:40257)
0621 05:20:17.223412 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.223414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.224104 (+ 690us) server_negotiation.cc:407] Connection header received
0621 05:20:17.224258 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.224259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.224282 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.224312 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.225326 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.226349 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.227385 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.227742 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.227845 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.229303 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.229304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.229304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.229312 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.231124 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.231524 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.231525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.231526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.231579 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.231816 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.231817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.231818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.231951 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.232374 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.232567 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.232644 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:20:17.235072 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.223772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.0.98.60:38175)
0621 05:20:17.223808 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.223809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.226893 (+ 3084us) server_negotiation.cc:407] Connection header received
0621 05:20:17.227916 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.227917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.227928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.227956 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.228668 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.229116 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.230149 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.230465 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.230536 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.231936 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.231937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.231938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.231945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.233659 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.234050 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.234051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.234051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.234122 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.234369 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.234370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.234370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.234483 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.234602 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.234891 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.234956 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:20:17.236924 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.224114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.0.98.62:33459)
0621 05:20:17.224171 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.224173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.227568 (+ 3395us) server_negotiation.cc:407] Connection header received
0621 05:20:17.227675 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.227676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.227687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.227717 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.228501 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.229228 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.232602 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.232900 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.232943 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.234329 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.234330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.234330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.234337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.235870 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.236155 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.236156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.236156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.236209 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.236426 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.236427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.236427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.236551 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.236631 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.236763 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.236837 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:20:17.239763 22871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:17.239980 22871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:17.241641 22803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [term 1 FOLLOWER]: Refusing update from remote peer e2c0b1c3e2c5463c870dbb19ef72d6ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:17.241963 22869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:17.243314 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c0b1c3e2c5463c870dbb19ef72d6ce. Latest consensus state: current_term: 1 leader_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.243407 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:17.244916 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.244997 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:17.245113 22872 mvcc.cc:205] Tried to move back new op lower bound from 7299150713819119616 to 7299150713763336192. Current Snapshot: MvccSnapshot[applied={T|T < 7299150713819119616}]
I20260621 05:20:17.245150 22873 mvcc.cc:205] Tried to move back new op lower bound from 7299150713819119616 to 7299150713763336192. Current Snapshot: MvccSnapshot[applied={T|T < 7299150713819119616}]
I20260621 05:20:17.245263 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.245337 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:17.248899 22871 catalog_manager.cc:1347] Generated new cluster ID: 42b6da6fdc7e43419f201f7e52bae942
I20260621 05:20:17.248956 22871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:17.256054 22754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attempt: 3
Jun 21 05:20:17 dist-test-slave-czm5 krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:20:17.264544 22871 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:17.265196 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.256307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:56558)
0621 05:20:17.256408 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:17.256419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:17.256453 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:17.256688 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:17.256689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:17.256871 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:17.256915 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.256917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.258263 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.258264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.259073 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:17.259076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.259567 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.259568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:17.259703 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.260172 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:17.260177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:17.261814 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:17.263973 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.263974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.263975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.264189 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.264450 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:17.264451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:17.264451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:17.264530 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:17.264896 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:17.264897 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:17.264950 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 05:20:17.265007 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:20:17.265101 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 05:20:17.265292 22871 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:17.265590 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:17.256519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.0.98.60:38175)
0621 05:20:17.256559 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.256561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.256568 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:20:17.256586 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.256587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.256595 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.256638 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.257036 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.257776 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.259167 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:17.259449 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:17.259508 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:17.261918 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:17.261919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:17.261920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:17.261926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:17.263874 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.264278 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.264279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.264279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.264349 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:17.264659 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:17.264660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:17.264660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:17.264793 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:17.264856 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:17.265416 (+ 560us) server_negotiation.cc:299] Negotiation successful
0621 05:20:17.265509 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:20:17.267117 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed: Bootstrap starting.
I20260621 05:20:17.267879 22754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:17.268244 22754 log.cc:824] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:17.269196 22871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce: Generated new TSK 0
I20260621 05:20:17.269376 22871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:17.272399 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed: No bootstrap required, opened a new log
I20260621 05:20:17.274457 22754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260621 05:20:17.274636 22754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1335d8cc12314ac8aaa14ff439d63fed, State: Initialized, Role: FOLLOWER
I20260621 05:20:17.274720 22754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260621 05:20:17.275041 22754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:17.275277 22728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1335d8cc12314ac8aaa14ff439d63fed"
I20260621 05:20:17.275707 22727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:17.276634 22727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [term 1 FOLLOWER]: Refusing update from remote peer e2c0b1c3e2c5463c870dbb19ef72d6ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:17.277513 22849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:20:17.277412 22889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:17.279889 22889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:17.280800 22728 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0b1c3e2c5463c870dbb19ef72d6ce in term 1.
I20260621 05:20:17.287634 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.287806 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:17.287894 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c0b1c3e2c5463c870dbb19ef72d6ce. Latest consensus state: current_term: 1 leader_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.287997 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:17.288055 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.288141 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:17.288879 22893 mvcc.cc:205] Tried to move back new op lower bound from 7299150713930100736 to 7299150713763336192. Current Snapshot: MvccSnapshot[applied={T|T < 7299150713819119616}]
W20260621 05:20:17.304553 22682 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.302980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.0.98.62:33459)
0621 05:20:17.303022 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.303025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.303883 (+ 858us) server_negotiation.cc:407] Connection header received
0621 05:20:17.303999 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.304001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.304012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.304042 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.304476 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55698: BlockingRecv error: recv got EOF from 127.0.0.1:55698 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:20:17.305303 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.303324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.0.98.61:40257)
0621 05:20:17.303359 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.303360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.304790 (+ 1430us) server_negotiation.cc:407] Connection header received
0621 05:20:17.304891 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.304893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.304904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.304939 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.305248 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56050: BlockingRecv error: recv got EOF from 127.0.0.1:56050 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260621 05:20:17.306001 22832 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.303596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.0.98.60:38175)
0621 05:20:17.303653 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.303654 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.305503 (+ 1849us) server_negotiation.cc:407] Connection header received
0621 05:20:17.305606 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.305607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.305617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.305645 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.305922 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56568: BlockingRecv error: recv got EOF from 127.0.0.1:56568 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260621 05:20:17.312312 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.308251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.0.98.61:40257)
0621 05:20:17.308283 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.308284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.311792 (+ 3508us) server_negotiation.cc:407] Connection header received
0621 05:20:17.311892 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.311893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.311909 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.311937 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.312248 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56058: BlockingRecv error: recv got EOF from 127.0.0.1:56058 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260621 05:20:17.315207 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.314482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.0.98.61:40257)
0621 05:20:17.314511 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.314512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.314736 (+ 224us) server_negotiation.cc:407] Connection header received
0621 05:20:17.314821 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.314822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.314848 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.314875 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.315162 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56066: BlockingRecv error: recv got EOF from 127.0.0.1:56066 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260621 05:20:17.318936 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.318253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.0.98.61:40257)
0621 05:20:17.318284 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.318286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.318484 (+ 198us) server_negotiation.cc:407] Connection header received
0621 05:20:17.318570 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.318571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.318594 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.318622 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.318896 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56070: BlockingRecv error: recv got EOF from 127.0.0.1:56070 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260621 05:20:17.324700 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.323977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.0.98.61:40257)
0621 05:20:17.324009 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.324011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.324209 (+ 198us) server_negotiation.cc:407] Connection header received
0621 05:20:17.324293 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.324294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.324303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.324337 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.324654 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56074: BlockingRecv error: recv got EOF from 127.0.0.1:56074 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260621 05:20:17.334714 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.333877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.0.98.61:40257)
0621 05:20:17.333915 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.333917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.334172 (+ 255us) server_negotiation.cc:407] Connection header received
0621 05:20:17.334269 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.334270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.334281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.334309 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.334660 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56080: BlockingRecv error: recv got EOF from 127.0.0.1:56080 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260621 05:20:17.353005 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.352051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.0.98.61:40257)
0621 05:20:17.352096 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.352098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.352393 (+ 295us) server_negotiation.cc:407] Connection header received
0621 05:20:17.352502 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.352503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.352515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.352547 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.352946 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56092: BlockingRecv error: recv got EOF from 127.0.0.1:56092 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:20:17.387141 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.386291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.0.98.61:40257)
0621 05:20:17.386329 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.386331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.386598 (+ 267us) server_negotiation.cc:407] Connection header received
0621 05:20:17.386693 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.386694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.386705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.386732 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.387075 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56100: BlockingRecv error: recv got EOF from 127.0.0.1:56100 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260621 05:20:17.453341 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.452625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.0.98.61:40257)
0621 05:20:17.452681 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.452684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.452813 (+ 129us) server_negotiation.cc:407] Connection header received
0621 05:20:17.452883 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.452884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.452896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.452955 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.453270 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56112: BlockingRecv error: recv got EOF from 127.0.0.1:56112 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260621 05:20:17.583199 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.582549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.0.98.61:40257)
0621 05:20:17.582611 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.582614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.582678 (+ 64us) server_negotiation.cc:407] Connection header received
0621 05:20:17.582755 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.582756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.582766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.582810 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.583116 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56114: BlockingRecv error: recv got EOF from 127.0.0.1:56114 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:20:17.814190 22952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0b1c3e2c5463c870dbb19ef72d6ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33459 } } peers { permanent_uuid: "1335d8cc12314ac8aaa14ff439d63fed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40257 } } peers { permanent_uuid: "e5cbf8c032bd4806839ff80fc8a41c5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260621 05:20:17.814347 22952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c0b1c3e2c5463c870dbb19ef72d6ce [sys.catalog]: This master's current role is: LEADER
W20260621 05:20:17.841224 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:17.840484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.0.98.61:40257)
0621 05:20:17.840563 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:17.840565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:17.840579 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:20:17.840700 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:17.840701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:17.840714 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:17.840786 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:17.841157 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56128: BlockingRecv error: recv got EOF from 127.0.0.1:56128 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:20:18.208606 22846 catalog_manager.cc:1259] Loaded cluster ID: 42b6da6fdc7e43419f201f7e52bae942
I20260621 05:20:18.208695 22846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d: loading cluster ID for follower catalog manager: success
I20260621 05:20:18.209520 22846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d: acquiring CA information for follower catalog manager: success
I20260621 05:20:18.209860 22846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5cbf8c032bd4806839ff80fc8a41c5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:20:18.280313 22889 catalog_manager.cc:1259] Loaded cluster ID: 42b6da6fdc7e43419f201f7e52bae942
I20260621 05:20:18.280390 22889 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed: loading cluster ID for follower catalog manager: success
I20260621 05:20:18.281217 22889 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed: acquiring CA information for follower catalog manager: success
I20260621 05:20:18.281524 22889 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1335d8cc12314ac8aaa14ff439d63fed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:20:18.356662 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:18.354314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.0.98.61:40257)
0621 05:20:18.354442 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:18.354444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:18.354453 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:18.354478 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:18.354479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:18.354525 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:18.354610 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:18.354749 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.355230 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.355992 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.356168 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.356222 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:18.356382 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:18.356488 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:18.356544 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:18.356578 (+ 34us) server_negotiation.cc:299] Negotiation successful
0621 05:20:18.356594 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260621 05:20:18.360834 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:18.358514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.0.98.60:38175)
0621 05:20:18.358615 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:18.358617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:18.358643 (+ 26us) server_negotiation.cc:407] Connection header received
0621 05:20:18.358689 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:18.358689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:18.358723 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:18.358792 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:18.358870 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.359371 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.360138 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.360337 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.360392 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:18.360508 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:18.360607 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:18.360662 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:18.360696 (+ 34us) server_negotiation.cc:299] Negotiation successful
0621 05:20:18.360715 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":54,"threads_started":1}
I20260621 05:20:18.365379 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:18.362609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.0.98.61:40257)
0621 05:20:18.362649 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:18.362651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:18.362725 (+ 74us) server_negotiation.cc:407] Connection header received
0621 05:20:18.362780 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:18.362780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:18.362791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:18.362840 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:18.362992 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.363475 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.364576 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.364879 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.364918 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:18.365097 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:18.365135 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:18.365199 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:18.365259 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 05:20:18.365280 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:20:18.365563 22978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:18.362354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.0.98.62:33459)
0621 05:20:18.362453 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:18.362455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:18.362505 (+ 50us) server_negotiation.cc:407] Connection header received
0621 05:20:18.362622 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:18.362623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:18.362689 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:18.362783 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:18.362952 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.363777 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.364566 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.364819 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.364883 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:18.365060 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:18.365170 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:18.365240 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:18.365451 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 05:20:18.365481 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
I20260621 05:20:18.365571 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:18.362589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.0.98.60:38175)
0621 05:20:18.362652 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:18.362654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:18.362750 (+ 96us) server_negotiation.cc:407] Connection header received
0621 05:20:18.362836 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:18.362836 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:18.362847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:18.362917 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:18.363040 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.363791 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.364788 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:18.365062 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:18.365120 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:18.365320 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:18.365349 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:18.365380 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:18.365492 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 05:20:18.365513 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:20:18.367327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22611
I20260621 05:20:18.372586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22686
I20260621 05:20:18.378867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22761
2026-06-21T05:20:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2751 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:18 dist-test-slave-czm5 krb5kdc[22982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:18 dist-test-slave-czm5 krb5kdc[22982](info): set up 2 sockets
Jun 21 05:20:18 dist-test-slave-czm5 krb5kdc[22982](info): commencing operation
krb5kdc: starting...
W20260621 05:20:19.395789 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/test-user.keytab.
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260621 05:20:19.506119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43531
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43531,127.0.98.61:44005,127.0.98.60:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:19.583386 22995 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:20:19.583585 22995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:19.583600 22995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:19.583636 22995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:19.584792 22995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:19.584831 22995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:19.584846 22995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:19.584858 22995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:19.584870 22995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:19.586300 22995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43531,127.0.98.61:44005,127.0.98.60:38537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:19.586477 22995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:19.586707 22995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:19.591554 23001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:19.591503 23000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:19.591586 22995 server_base.cc:1034] running on GCE node
W20260621 05:20:19.591507 23003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:19.592025 22995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:19.592330 22995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:19.593488 22995 hybrid_clock.cc:648] HybridClock initialized: now 1782019219593479 us; error 35 us; skew 500 ppm
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:19.596032 22995 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260621 05:20:19.596447 22995 webserver.cc:460] Webserver started at http://127.0.98.62:37015/ using document root <none> and password file <none>
I20260621 05:20:19.596658 22995 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:19.596721 22995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:19.596853 22995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:19.597913 22995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/data/instance:
uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "9d498ceeec46a7cb54e6e355d6ba2917"
server_key_iv: "3c3fba4934a7b2ecd996f233a157e9fb"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.598305 22995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/wal/instance:
uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "9d498ceeec46a7cb54e6e355d6ba2917"
server_key_iv: "3c3fba4934a7b2ecd996f233a157e9fb"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.599534 22995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:20:19.600237 23010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:19.600404 22995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:19.600493 22995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/wal
uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "9d498ceeec46a7cb54e6e355d6ba2917"
server_key_iv: "3c3fba4934a7b2ecd996f233a157e9fb"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.600570 22995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:19.613270 22995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:19.615443 22995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:19.615597 22995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:19.619371 22995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43531
I20260621 05:20:19.619426 23062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43531 every 8 connection(s)
I20260621 05:20:19.619793 22995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:19.620422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 22995
I20260621 05:20:19.620549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-0/wal/instance
I20260621 05:20:19.620741 392 external_mini_cluster.cc:1428] Setting key b763a6c4c66c8de17eccc97ffc90033d
I20260621 05:20:19.620923 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:19.622929 23063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:20:19.634348 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.623698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.0.98.62:43531)
0621 05:20:19.623867 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.623876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.623986 (+ 110us) server_negotiation.cc:407] Connection header received
0621 05:20:19.624550 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.624552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.624578 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.624674 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.625204 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.625983 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.626803 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.627094 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.627203 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.630946 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.630954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.630955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.630963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.632780 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.633233 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.633236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.633236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.633310 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.633566 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.633566 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.633567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.633796 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.633859 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.634087 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.634190 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260621 05:20:19.634547 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.623644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43531 (local address 127.0.0.1:32984)
0621 05:20:19.623847 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.623889 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.624404 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.624680 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.624682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.624951 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.625111 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.625116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.626073 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.626074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.626713 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.626715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.627199 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.627200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.627259 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.627766 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:19.627774 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:19.630859 (+ 3085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:19.632864 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.632871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.632873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.633152 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.633374 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.633375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.633376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.633492 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.633866 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:19.633873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:19.633957 (+ 84us) client_negotiation.cc:769] Sending connection context
0621 05:20:19.634063 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 05:20:19.634287 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":48,"thread_start_us":114,"threads_started":1}
I20260621 05:20:19.635357 23063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:19.635773 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.635603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55878)
0621 05:20:19.635698 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.635750 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260621 05:20:19.635886 23013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
I20260621 05:20:19.636294 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.636216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55886)
0621 05:20:19.636251 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.636276 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260621 05:20:19.636406 23063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44005: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260621 05:20:19.644510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44005
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43531,127.0.98.61:44005,127.0.98.60:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:20:19.685621 23063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } attempt: 1
I20260621 05:20:19.686088 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.685960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55892)
0621 05:20:19.686025 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.686061 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 05:20:19.686439 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.686347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55898)
0621 05:20:19.686393 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.686417 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:20:19.686537 23063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44005: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
W20260621 05:20:19.722834 23070 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:20:19.723075 23070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:19.723105 23070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:19.723143 23070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:19.724300 23070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:19.724363 23070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:19.724386 23070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:19.724408 23070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:19.724447 23070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:19.725836 23070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43531,127.0.98.61:44005,127.0.98.60:38537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:19.726097 23070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:19.726302 23070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:19.731040 23075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:19.731087 23076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:19.731350 23070 server_base.cc:1034] running on GCE node
W20260621 05:20:19.731057 23078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:19.731664 23070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:19.731983 23070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:19.733130 23070 hybrid_clock.cc:648] HybridClock initialized: now 1782019219733094 us; error 56 us; skew 500 ppm
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:19.735607 23070 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260621 05:20:19.736053 23070 webserver.cc:460] Webserver started at http://127.0.98.61:41285/ using document root <none> and password file <none>
I20260621 05:20:19.736238 23070 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:19.736275 23070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:19.736362 23070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:19.737331 23070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/data/instance:
uuid: "a00124f7e5984315a936a6ac8d0d91e2"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "11ce69da88573909ffc5c2fe26323d4f"
server_key_iv: "a34b30c41f79331fdab977ed9d0c6568"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.737622 23070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/wal/instance:
uuid: "a00124f7e5984315a936a6ac8d0d91e2"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "11ce69da88573909ffc5c2fe26323d4f"
server_key_iv: "a34b30c41f79331fdab977ed9d0c6568"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.738880 23070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:20:19.739496 23085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:19.739688 23070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:19.739773 23070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/wal
uuid: "a00124f7e5984315a936a6ac8d0d91e2"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "11ce69da88573909ffc5c2fe26323d4f"
server_key_iv: "a34b30c41f79331fdab977ed9d0c6568"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.739845 23070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:19.754673 23063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } attempt: 2
I20260621 05:20:19.755033 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.754920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55914)
0621 05:20:19.754975 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.755007 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260621 05:20:19.755193 23070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:19.755364 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.755266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55924)
0621 05:20:19.755319 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.755341 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260621 05:20:19.755472 23063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44005: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44005: connect: Connection refused (error 111)
I20260621 05:20:19.757041 23070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:19.757201 23070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:19.760962 23070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44005
I20260621 05:20:19.760990 23137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44005 every 8 connection(s)
I20260621 05:20:19.761329 23070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/data/info.pb
I20260621 05:20:19.762260 23138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:19.763806 23138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:20:19.768978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 23070
I20260621 05:20:19.769071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-1/wal/instance
I20260621 05:20:19.769258 392 external_mini_cluster.cc:1428] Setting key 3be443f0a27d1323d5efe8d40c181765
I20260621 05:20:19.772166 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.764392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43531 (local address 127.0.0.1:32992)
0621 05:20:19.764550 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.764583 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.765021 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.765236 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.765239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.765462 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.765641 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.765644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.766246 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.766247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.766988 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.766989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.767307 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.767308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.767403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.767824 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:19.767840 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:19.769330 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:19.770840 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.770847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.770849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.771062 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.771286 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.771286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.771287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.771392 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.771712 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:19.771718 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:19.771828 (+ 110us) client_negotiation.cc:769] Sending connection context
0621 05:20:19.771914 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 05:20:19.772021 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260621 05:20:19.772181 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.764421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.0.98.62:43531)
0621 05:20:19.764487 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.764489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.764656 (+ 167us) server_negotiation.cc:407] Connection header received
0621 05:20:19.765111 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.765112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.765124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.765165 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.765725 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.766180 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.767052 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.767236 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.767281 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.769429 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.769430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.769430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.769436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.770758 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.771137 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.771138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.771138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.771205 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.771469 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.771470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.771470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.771613 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.771710 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.771955 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.772047 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:20:19.772709 23138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260621 05:20:19.779474 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.772957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55934)
0621 05:20:19.773013 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.773022 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.773058 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.773304 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.773304 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.773436 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.773475 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.773476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.774292 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.774293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.774766 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.774767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.775206 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.775207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.775273 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.775642 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:19.775647 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:19.776804 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:19.778523 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.778524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.778524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.778670 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.778895 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.778896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.778896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.778956 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.779236 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:19.779237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:19.779282 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 05:20:19.779350 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:20:19.779415 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 05:20:19.779764 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.773029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.0.98.61:44005)
0621 05:20:19.773181 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.773185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.773200 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:20:19.773215 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.773216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.773226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.773304 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.773550 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.774203 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.774851 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.775130 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.775199 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.776875 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.776881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.776882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.776888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.778452 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.778730 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.778732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.778732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.778777 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.779030 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.779030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.779031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.779172 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.779233 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.779407 (+ 174us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.779494 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260621 05:20:19.780318 23138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:19.780614 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.780532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50018)
0621 05:20:19.780561 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.780599 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260621 05:20:19.780755 23088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
I20260621 05:20:19.781138 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.781064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50034)
0621 05:20:19.781101 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.781124 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260621 05:20:19.781245 23138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38537: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260621 05:20:19.792577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaWoYkf/build/release/bin/kudu
/tmp/dist-test-taskaWoYkf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38537
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43531,127.0.98.61:44005,127.0.98.60:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:20:19.830480 23138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } attempt: 1
I20260621 05:20:19.831200 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.831045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50042)
0621 05:20:19.831113 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.831157 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 05:20:19.831575 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.831484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50058)
0621 05:20:19.831525 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.831552 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 05:20:19.831707 23138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38537: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
I20260621 05:20:19.846596 23063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } attempt: 3
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260621 05:20:19.853714 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.846928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55938)
0621 05:20:19.846970 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.846985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.847049 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.847238 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.847239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.847404 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.847465 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.847467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.848180 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.848181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.848691 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.848693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.849075 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.849076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.849160 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.849601 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:19.849606 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:19.851030 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:19.852701 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.852702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.852702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.852872 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.853078 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.853079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.853079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.853141 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.853377 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:19.853378 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:19.853423 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 05:20:19.853504 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:20:19.853612 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260621 05:20:19.853714 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.846975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.0.98.61:44005)
0621 05:20:19.847030 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.847032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.847044 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:20:19.847137 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.847138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.847149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.847233 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.847544 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.848087 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.848785 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.848994 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.849063 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.851122 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.851123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.851123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.851130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.852592 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.852953 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.852954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.852954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.853013 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.853201 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.853202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.853202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.853315 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.853385 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.853527 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.853606 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:20:19.854100 23063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:19.854430 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.854337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50062)
0621 05:20:19.854381 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.854407 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:20:19.854749 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.854695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50068)
0621 05:20:19.854718 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.854738 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260621 05:20:19.854847 23063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38537: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
W20260621 05:20:19.871904 23145 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:20:19.872093 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:19.872110 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:19.872151 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:19.873255 23145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:19.873294 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:19.873308 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:19.873358 23145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:19.873376 23145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:19.874758 23145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43531,127.0.98.61:44005,127.0.98.60:38537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:19.874976 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:19.875206 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:19.879794 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:19.879859 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:19.879864 23145 server_base.cc:1034] running on GCE node
W20260621 05:20:19.879853 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:19.880244 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:19.880533 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:19.881695 23145 hybrid_clock.cc:648] HybridClock initialized: now 1782019219881671 us; error 33 us; skew 500 ppm
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:19.884064 23145 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260621 05:20:19.884507 23145 webserver.cc:460] Webserver started at http://127.0.98.60:39433/ using document root <none> and password file <none>
I20260621 05:20:19.884711 23145 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:19.884752 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:19.884836 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:19.885808 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/data/instance:
uuid: "a1c9df10468a4f52ae4d1444e0439cba"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "d94d01fb34b7869c5f911b3c9c05ec04"
server_key_iv: "941bbce7f3d8f188a5440f8d7c7c9da6"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.885993 23063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } attempt: 1
I20260621 05:20:19.886183 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/wal/instance:
uuid: "a1c9df10468a4f52ae4d1444e0439cba"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "d94d01fb34b7869c5f911b3c9c05ec04"
server_key_iv: "941bbce7f3d8f188a5440f8d7c7c9da6"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.886381 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.886270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50084)
0621 05:20:19.886321 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.886349 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260621 05:20:19.886677 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.886588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50090)
0621 05:20:19.886633 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.886658 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260621 05:20:19.886775 23063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38537: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
I20260621 05:20:19.887658 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:19.888444 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:19.888612 23145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:19.888710 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/wal
uuid: "a1c9df10468a4f52ae4d1444e0439cba"
format_stamp: "Formatted at 2026-06-21 05:20:19 on dist-test-slave-czm5"
server_key: "d94d01fb34b7869c5f911b3c9c05ec04"
server_key_iv: "941bbce7f3d8f188a5440f8d7c7c9da6"
server_key_version: "encryptionkey@0"
I20260621 05:20:19.888783 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:19.899835 23138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } attempt: 2
I20260621 05:20:19.900193 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.900077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50104)
0621 05:20:19.900135 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.900170 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 05:20:19.900557 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.900481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50112)
0621 05:20:19.900519 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.900542 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:20:19.900667 23138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38537: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38537: connect: Connection refused (error 111)
I20260621 05:20:19.908167 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:19.909803 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:19.909969 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:19.913847 23145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38537
I20260621 05:20:19.913879 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38537 every 8 connection(s)
I20260621 05:20:19.914222 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/data/info.pb
I20260621 05:20:19.915144 23213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:19.916800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaWoYkf/build/release/bin/kudu as pid 23145
I20260621 05:20:19.916882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaWoYkf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018933478224-392-0/minicluster-data/master-2/wal/instance
I20260621 05:20:19.916713 23213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:19.917085 392 external_mini_cluster.cc:1428] Setting key f3672bd11e9dacb675bb3116b62fc62e
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:20:19.925873 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.917433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.0.98.62:43531)
0621 05:20:19.917485 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.917487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.917839 (+ 352us) server_negotiation.cc:407] Connection header received
0621 05:20:19.918158 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.918159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.918172 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.918231 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.918775 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.919227 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.920141 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.920332 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.920424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.922627 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.922628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.922628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.922635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.924455 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.924872 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.924873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.924873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.924933 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.925211 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.925212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.925212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.925331 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.925404 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.925711 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.925788 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:20:19.925988 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.917422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43531 (local address 127.0.0.1:32994)
0621 05:20:19.917740 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.917768 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.918073 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.918252 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.918254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.918525 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.918672 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.918676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.919301 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.919302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.920059 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.920061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.920451 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.920452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.920564 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.921032 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:19.921042 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:19.922540 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:19.924533 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.924538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.924540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.924804 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.924997 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.924997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.924997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.925137 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.925401 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:19.925405 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:19.925556 (+ 151us) client_negotiation.cc:769] Sending connection context
0621 05:20:19.925658 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 05:20:19.925764 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":79,"threads_started":1}
I20260621 05:20:19.926409 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.917483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.0.98.60:38537)
0621 05:20:19.917644 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.917651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.917674 (+ 23us) server_negotiation.cc:407] Connection header received
0621 05:20:19.918059 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.918065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.918083 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.918193 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.918821 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.919561 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.920371 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.920661 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.920735 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.923201 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.923209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.923209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.923216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.925013 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.925356 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.925358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.925359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.925413 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.925595 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.925596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.925596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.925758 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.925890 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.926245 (+ 355us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.926316 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260621 05:20:19.926527 23213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260621 05:20:19.933655 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.926791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44005 (local address 127.0.0.1:55940)
0621 05:20:19.926838 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.926849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.926889 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.927023 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.927023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.927143 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.927182 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.927183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.928029 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.928030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.928697 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.928699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.929151 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.929152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.929195 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.929569 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:19.929574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:19.931010 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:19.932668 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.932669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.932669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.932849 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.933049 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.933050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.933051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.933111 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.933353 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:19.933354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:19.933392 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 05:20:19.933465 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:20:19.933556 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 05:20:19.933682 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.926848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.0.98.61:44005)
0621 05:20:19.926891 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.926893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.926934 (+ 41us) server_negotiation.cc:407] Connection header received
0621 05:20:19.926958 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.926959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.926968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.927017 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.927249 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.927967 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.928762 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.929083 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.929148 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.931112 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.931114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.931114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.931121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.932579 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.932930 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.932931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.932931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.932988 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.933171 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.933172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.933173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.933281 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.933348 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.933503 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.933573 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:20:19.933986 23213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:19.934887 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.927222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.0.98.62:43531)
0621 05:20:19.927255 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.927257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.927262 (+ 5us) server_negotiation.cc:407] Connection header received
0621 05:20:19.927280 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.927281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.927288 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.927330 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.927864 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.928482 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.929337 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.929529 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.929605 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.932186 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.932187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.932187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.932194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.933560 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.934102 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.934103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.934103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.934160 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.934352 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.934354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.934354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.934454 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.934628 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.934661 (+ 33us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.934739 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260621 05:20:19.940062 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.934253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50128)
0621 05:20:19.934283 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.934292 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.934328 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.934443 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.934443 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.934561 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.934581 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.934582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.935226 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.935227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.935801 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.935803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.936145 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.936146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.936191 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.936524 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:19.936528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:19.937751 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:19.939097 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.939098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.939099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.939251 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.939434 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.939435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.939436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.939483 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.939769 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:19.939770 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:19.939803 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 05:20:19.939870 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:20:19.939958 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260621 05:20:19.940110 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.934282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.0.98.60:38537)
0621 05:20:19.934310 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.934312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.934319 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:20:19.934382 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.934383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.934390 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.934440 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.934650 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.935151 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.935869 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.936065 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.936146 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.937823 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.937824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.937824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.937830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.939008 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.939326 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.939327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.939327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.939371 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.939553 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.939554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.939554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.939706 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.939778 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.939890 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.940004 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":17}
I20260621 05:20:19.941270 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba: Bootstrap starting.
I20260621 05:20:19.942093 23213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:19.942379 23213 log.cc:824] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:19.942440 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.935425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.98.61:44005)
0621 05:20:19.935463 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.935466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.935484 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:20:19.935501 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.935502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.935509 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.935551 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.935997 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.936537 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.937366 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.937576 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.937654 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.939830 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.939831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.939831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.939838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.941371 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.941685 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.941686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.941686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.941739 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.941919 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.941920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.941921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.942020 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.942181 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.942224 (+ 43us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.942292 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:20:19.943331 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba: No bootstrap required, opened a new log
I20260621 05:20:19.943850 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:19.944890 23213 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } }
I20260621 05:20:19.945019 23213 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c9df10468a4f52ae4d1444e0439cba, State: Initialized, Role: FOLLOWER
I20260621 05:20:19.945103 23213 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } }
I20260621 05:20:19.945405 23213 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:19.945369 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:19.945465 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:19.944715 23037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32998
W20260621 05:20:19.946919 23230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:19.946972 23230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:19.947180 23145 master_runner.cc:186] Error getting master registration for 127.0.98.62:43531: OK, instance_id { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" instance_seqno: 1782019219618350 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:20:19.949213 23145 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:20:19.961982 23063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } attempt: 2
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:20:19.968607 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.962356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.0.98.60:38537)
0621 05:20:19.962410 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.962412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.962421 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:19.962470 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.962471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.962482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.962568 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.962854 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.963311 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.964020 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.964209 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.964294 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.966057 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.966058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.966059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.966066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.967429 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.967841 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.967842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.967843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.967896 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.968087 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.968088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.968088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.968223 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.968293 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:19.968443 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 05:20:19.968512 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:20:19.968607 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.962302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50144)
0621 05:20:19.962354 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.962364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.962410 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.962574 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.962575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.962739 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.962779 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.962781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.963396 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.963397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.963942 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.963944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.964304 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.964306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.964361 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.964799 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:19.964804 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:19.965974 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:19.967526 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.967527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.967528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.967762 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.967960 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:19.967961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:19.967962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:19.968021 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:19.968296 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:19.968297 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:19.968341 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:20:19.968419 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:20:19.968513 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:20:19.970248 23063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866: Bootstrap starting.
I20260621 05:20:19.970850 23063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:19.971138 23063 log.cc:824] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:19.971930 23063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866: No bootstrap required, opened a new log
I20260621 05:20:19.972919 23063 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } }
I20260621 05:20:19.973049 23063 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac0feb56ebc4cbc9dcf3ea2f1494866, State: Initialized, Role: FOLLOWER
I20260621 05:20:19.973143 23063 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } }
I20260621 05:20:19.973440 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:19.973485 23063 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:19.973529 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:19.973639 23037 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 05:20:19.973734 23037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:19.974499 23037 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } }
W20260621 05:20:19.975297 23244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:19.975387 23244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:19.975368 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a1c9df10468a4f52ae4d1444e0439cba"
I20260621 05:20:19.975497 23187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:19.975654 23037 leader_election.cc:288] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [CANDIDATE]: Term 1 election: Requested vote from peers a00124f7e5984315a936a6ac8d0d91e2 (127.0.98.61:44005), a1c9df10468a4f52ae4d1444e0439cba (127.0.98.60:38537)
I20260621 05:20:19.976346 23187 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac0feb56ebc4cbc9dcf3ea2f1494866 in term 1.
I20260621 05:20:19.977432 23013 leader_election.cc:302] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1c9df10468a4f52ae4d1444e0439cba, aac0feb56ebc4cbc9dcf3ea2f1494866; no voters:
I20260621 05:20:19.977538 23233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:19.977677 23233 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [term 1 LEADER]: Becoming Leader. State: Replica: aac0feb56ebc4cbc9dcf3ea2f1494866, State: Running, Role: LEADER
I20260621 05:20:19.977751 23233 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } }
I20260621 05:20:19.978102 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac0feb56ebc4cbc9dcf3ea2f1494866. Latest consensus state: current_term: 1 leader_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:19.978183 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:19.976087 22995 master_runner.cc:186] Error getting master registration for 127.0.98.61:44005: OK, instance_id { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" instance_seqno: 1782019219759970 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:20:19.978829 22995 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:20:19.979067 23252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:19.979249 23252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:19.981230 23187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [term 1 FOLLOWER]: Refusing update from remote peer aac0feb56ebc4cbc9dcf3ea2f1494866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:19.981756 23233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:19.983245 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac0feb56ebc4cbc9dcf3ea2f1494866. Latest consensus state: current_term: 1 leader_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:19.983332 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:19.985639 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:19.985728 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:19.986261 23253 mvcc.cc:205] Tried to move back new op lower bound from 7299150725039513600 to 7299150725029457920. Current Snapshot: MvccSnapshot[applied={T|T < 7299150725039513600}]
I20260621 05:20:19.986714 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:19.986794 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:19.987525 23257 mvcc.cc:205] Tried to move back new op lower bound from 7299150725039513600 to 7299150725029457920. Current Snapshot: MvccSnapshot[applied={T|T < 7299150725039513600}]
I20260621 05:20:19.989538 23252 catalog_manager.cc:1347] Generated new cluster ID: 48f4e5cb954f49408dcc0b76ceae6033
I20260621 05:20:19.989586 23252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:19.991788 23138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } attempt: 3
I20260621 05:20:20.000288 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.980737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.0.98.62:43531)
0621 05:20:19.980771 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.980773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.991759 (+ 10986us) server_negotiation.cc:407] Connection header received
0621 05:20:19.992234 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.992235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.992245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.992308 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.992848 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.993576 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.994495 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.994813 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.994887 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.996047 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.996048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.996048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.996055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.997845 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.998240 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.998241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.998241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.998303 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:19.999574 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:19.999575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:19.999575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:19.999714 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:19.999803 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:20.000098 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 05:20:20.000194 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 21 05:20:19 dist-test-slave-czm5 krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260621 05:20:20.004938 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.980177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.0.98.61:44005)
0621 05:20:19.980220 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.980222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.981011 (+ 789us) server_negotiation.cc:407] Connection header received
0621 05:20:19.981105 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.981106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.981117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.981146 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.982815 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.983565 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.986458 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.986794 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.986835 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:19.991684 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:19.991686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:19.991686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:19.991694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:19.993459 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:20.002794 (+ 9335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:20.002795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:20.002795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:20.002859 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:20.003605 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:20.003605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:20.003606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:20.003785 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:20.003930 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:20.004763 (+ 833us) server_negotiation.cc:299] Negotiation successful
0621 05:20:20.004842 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:20:20.008388 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.980475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.0.98.60:38537)
0621 05:20:19.980512 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.980514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.981965 (+ 1451us) server_negotiation.cc:407] Connection header received
0621 05:20:19.982054 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.982055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.982064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.982091 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.984713 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.985427 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.987180 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.987457 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.987494 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:20.003690 (+ 16196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:20.003691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:20.003691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:20.003698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:20.005679 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:20.006428 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:20.006429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:20.006429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:20.006488 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:20.007848 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:20.007849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:20.007849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:20.007982 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:20.008081 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:20.008210 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 05:20:20.008298 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:20:20.010700 23252 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:20.011375 23252 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:20.013964 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.992046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.0.98.60:38537)
0621 05:20:19.994487 (+ 2441us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:19.994489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:19.994496 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:20:19.994513 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:19.994514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:19.994524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:19.994832 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:19.995075 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.996569 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.999852 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:20.000167 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:20.000417 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:20.001982 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:20.001983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:20.001984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:20.001990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:20.012812 (+ 10822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:20.013215 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:20.013216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:20.013216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:20.013275 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:20.013440 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:20.013441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:20.013441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:20.013551 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:20.013597 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:20.013796 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 05:20:20.013866 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260621 05:20:20.013967 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:19.992001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38537 (local address 127.0.0.1:50170)
0621 05:20:19.992051 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:19.992060 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:19.992096 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:19.994607 (+ 2511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:19.994608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:19.994741 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:19.994776 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.994777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:19.998250 (+ 3473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:19.998251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:19.999365 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:19.999367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:20.000242 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:20.000243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:20.000297 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:20.000643 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:20.000647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:20.001899 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:20.012905 (+ 11006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:20.012906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:20.012907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:20.013144 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:20.013336 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:20.013337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:20.013337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:20.013388 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:20.013628 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:20.013628 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:20.013678 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 05:20:20.013773 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:20:20.013871 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 05:20:20.015419 23138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2: Bootstrap starting.
I20260621 05:20:20.016153 23138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:20.016439 23138 log.cc:824] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:20.017490 23138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2: No bootstrap required, opened a new log
I20260621 05:20:20.019447 23138 raft_consensus.cc:348] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } }
I20260621 05:20:20.019575 23138 raft_consensus.cc:729] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00124f7e5984315a936a6ac8d0d91e2, State: Initialized, Role: FOLLOWER
I20260621 05:20:20.020198 23138 consensus_queue.cc:260] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } }
I20260621 05:20:20.020853 23112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a00124f7e5984315a936a6ac8d0d91e2"
I20260621 05:20:20.020953 23112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:20.021785 23112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac0feb56ebc4cbc9dcf3ea2f1494866 in term 1.
I20260621 05:20:20.022032 23111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [term 1 FOLLOWER]: Refusing update from remote peer aac0feb56ebc4cbc9dcf3ea2f1494866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:20.022234 23138 sys_catalog.cc:564] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:20.022284 23260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:20:20.025943 23279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:20.026038 23279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:20.026489 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:20.026574 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:20.026641 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac0feb56ebc4cbc9dcf3ea2f1494866. Latest consensus state: current_term: 1 leader_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:20.026700 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:20.026752 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:20.026796 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:20.032133 23273 mvcc.cc:205] Tried to move back new op lower bound from 7299150725155299328 to 7299150725029457920. Current Snapshot: MvccSnapshot[applied={T|T < 7299150725039513600}]
I20260621 05:20:20.035878 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac0feb56ebc4cbc9dcf3ea2f1494866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43531 } } peers { permanent_uuid: "a00124f7e5984315a936a6ac8d0d91e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44005 } } peers { permanent_uuid: "a1c9df10468a4f52ae4d1444e0439cba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38537 } } }
I20260621 05:20:20.035972 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:20.038069 23252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac0feb56ebc4cbc9dcf3ea2f1494866: Generated new TSK 0
I20260621 05:20:20.038257 23252 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:20:20.063753 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.062010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.0.98.61:44005)
0621 05:20:20.062064 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.062067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.063005 (+ 938us) server_negotiation.cc:407] Connection header received
0621 05:20:20.063122 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.063123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.063135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.063166 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.063659 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55952: BlockingRecv error: recv got EOF from 127.0.0.1:55952 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260621 05:20:20.064522 23262 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.062396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.0.98.60:38537)
0621 05:20:20.062442 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.062444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.064010 (+ 1566us) server_negotiation.cc:407] Connection header received
0621 05:20:20.064079 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.064080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.064093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.064160 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.064463 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50182: BlockingRecv error: recv got EOF from 127.0.0.1:50182 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260621 05:20:20.064656 23067 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.062724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.0.98.62:43531)
0621 05:20:20.062763 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.062765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.064162 (+ 1397us) server_negotiation.cc:407] Connection header received
0621 05:20:20.064256 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.064257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.064267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.064296 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.064591 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33012: BlockingRecv error: recv got EOF from 127.0.0.1:33012 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:20:20.069916 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.069131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.0.98.61:44005)
0621 05:20:20.069166 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.069168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.069388 (+ 220us) server_negotiation.cc:407] Connection header received
0621 05:20:20.069497 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.069498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.069515 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.069545 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.069867 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55962: BlockingRecv error: recv got EOF from 127.0.0.1:55962 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260621 05:20:20.072899 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.072098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.0.98.61:44005)
0621 05:20:20.072131 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.072132 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.072371 (+ 239us) server_negotiation.cc:407] Connection header received
0621 05:20:20.072465 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.072466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.072485 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.072514 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.072854 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55966: BlockingRecv error: recv got EOF from 127.0.0.1:55966 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260621 05:20:20.076710 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.075919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.0.98.61:44005)
0621 05:20:20.075952 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.075953 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.076192 (+ 239us) server_negotiation.cc:407] Connection header received
0621 05:20:20.076286 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.076286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.076296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.076326 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.076651 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55970: BlockingRecv error: recv got EOF from 127.0.0.1:55970 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260621 05:20:20.082626 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.081806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.0.98.61:44005)
0621 05:20:20.081843 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.081845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.082089 (+ 244us) server_negotiation.cc:407] Connection header received
0621 05:20:20.082191 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.082191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.082201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.082231 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.082570 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55976: BlockingRecv error: recv got EOF from 127.0.0.1:55976 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:20:20.092562 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.091719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.98.61:44005)
0621 05:20:20.091787 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.091789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.092054 (+ 265us) server_negotiation.cc:407] Connection header received
0621 05:20:20.092143 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.092144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.092155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.092184 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.092497 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55980: BlockingRecv error: recv got EOF from 127.0.0.1:55980 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 05:20:20.110585 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.109719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.0.98.61:44005)
0621 05:20:20.109757 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.109759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.110025 (+ 266us) server_negotiation.cc:407] Connection header received
0621 05:20:20.110126 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.110127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.110139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.110169 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.110532 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55982: BlockingRecv error: recv got EOF from 127.0.0.1:55982 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:20:20.144553 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.143845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.0.98.61:44005)
0621 05:20:20.143891 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.143893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.144031 (+ 138us) server_negotiation.cc:407] Connection header received
0621 05:20:20.144109 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.144110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.144120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.144180 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.144483 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55998: BlockingRecv error: recv got EOF from 127.0.0.1:55998 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:20:20.210357 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.209719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.0.98.61:44005)
0621 05:20:20.209812 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.209814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.209827 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:20:20.209904 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.209905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.209915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.210007 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.210284 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56004: BlockingRecv error: recv got EOF from 127.0.0.1:56004 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 05:20:20.340197 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.339476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.0.98.61:44005)
0621 05:20:20.339545 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.339547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.339659 (+ 112us) server_negotiation.cc:407] Connection header received
0621 05:20:20.339744 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.339745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.339756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.339808 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.340138 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56016: BlockingRecv error: recv got EOF from 127.0.0.1:56016 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260621 05:20:20.598130 23142 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:20.597371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.0.98.61:44005)
0621 05:20:20.597438 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:20.597441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:20.597513 (+ 72us) server_negotiation.cc:407] Connection header received
0621 05:20:20.597626 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:20.597626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:20.597637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:20.597700 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:20.598061 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56022: BlockingRecv error: recv got EOF from 127.0.0.1:56022 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:20:20.947391 23230 catalog_manager.cc:1259] Loaded cluster ID: 48f4e5cb954f49408dcc0b76ceae6033
I20260621 05:20:20.947469 23230 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba: loading cluster ID for follower catalog manager: success
I20260621 05:20:20.948287 23230 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba: acquiring CA information for follower catalog manager: success
I20260621 05:20:20.948549 23230 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a1c9df10468a4f52ae4d1444e0439cba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:20:21.026566 23279 catalog_manager.cc:1259] Loaded cluster ID: 48f4e5cb954f49408dcc0b76ceae6033
I20260621 05:20:21.026674 23279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2: loading cluster ID for follower catalog manager: success
I20260621 05:20:21.027498 23279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2: acquiring CA information for follower catalog manager: success
I20260621 05:20:21.027841 23279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a00124f7e5984315a936a6ac8d0d91e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:20:21.114145 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:21.111421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.0.98.61:44005)
0621 05:20:21.111536 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:21.111539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:21.111611 (+ 72us) server_negotiation.cc:407] Connection header received
0621 05:20:21.111708 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:21.111709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:21.111773 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:21.111864 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:21.112016 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.112508 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.113314 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.113518 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.113589 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:21.113780 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:21.113906 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:21.113993 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:21.114011 (+ 18us) server_negotiation.cc:299] Negotiation successful
0621 05:20:21.114030 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260621 05:20:21.118445 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:21.115935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.0.98.60:38537)
0621 05:20:21.116053 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:21.116055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:21.116063 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:20:21.116172 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:21.116173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:21.116223 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:21.116301 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:21.116430 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.116913 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.117653 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.117870 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.117948 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:21.118102 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:21.118208 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:21.118297 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:21.118337 (+ 40us) server_negotiation.cc:299] Negotiation successful
0621 05:20:21.118357 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":46,"threads_started":1}
I20260621 05:20:21.122503 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:21.120070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.0.98.61:44005)
0621 05:20:21.120112 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:21.120114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:21.120206 (+ 92us) server_negotiation.cc:407] Connection header received
0621 05:20:21.120313 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:21.120314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:21.120324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:21.120367 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:21.120468 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.120905 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.121797 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.122028 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.122090 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:21.122250 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:21.122279 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:21.122323 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:21.122401 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:20:21.122437 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:20:21.123194 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:21.120285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.0.98.60:38537)
0621 05:20:21.120317 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:21.120318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:21.120594 (+ 276us) server_negotiation.cc:407] Connection header received
0621 05:20:21.120654 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:21.120655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:21.120663 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:21.120724 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:21.120833 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.121400 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.122338 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.122763 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.122832 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:21.122953 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:21.122983 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:21.123034 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:21.123097 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 05:20:21.123113 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:20:21.123754 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:21.120068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.0.98.62:43531)
0621 05:20:21.120176 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:21.120178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:21.120189 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:21.120343 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:21.120345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:21.120396 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:21.120477 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:21.121142 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.121811 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.122779 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:21.123108 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:21.123173 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:21.123376 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:21.123478 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:21.123538 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:21.123596 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 05:20:21.123646 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260621 05:20:21.128738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 22995
I20260621 05:20:21.134161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 23070
I20260621 05:20:21.138760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaWoYkf/build/release/bin/kudu with pid 23145
2026-06-21T05:20:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2759 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5510 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287667 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS